Multimedia platform for recording and/or reproducing music synchronously with visual images

ABSTRACT

A multimedia platform records a performance on a keyboard synchronously with a picture by periodically regulating an internal clock, which is indicative of the lapse of time, with time codes inserted into the set of video data codes representative of the picture, and reproduces the performance through an automatic player piano also synchronously with the picture by periodically regulating the internal clock with the time codes, whereby the user enjoys himself or herself in the performance as if he or she feels himself or herself performing in a convert hall.

FIELD OF THE INVENTION

This invention relates to a multimedia platform and, more particularly,to a multimedia platform for synchronously recording and reproducingvisual images and music and a recorder/reproducer incorporated therein.

DESCRIPTION OF THE RELATED ART

One of the desires of amateur music players is to play in a concerthall. However, it is a dream for most of the amateur music players. Anamateur music player projects a picture on a monitor screen such as, forexample, a liquid crystal display panel during his or her performance,and enjoys himself or herself by performing a piece of music in avirtual concert hall. While he or she is performing the piece of music,a video cassette player may read out the video data from a videotapecassette for reproducing the picture on the monitor screen. If anamateur music player wishes to record and, thereafter, reproduce his orher performance, he or she fingers the piece of music on a musicalinstrument with an automatic recording/playing system. The automaticrecording/playing system, by way of example, converts the key actions toMIDI (Musical Instrument Digital Interface) data codes, and stores themin a suitable information storage medium such as a floppy disc. The MIDIdata codes are broken down into event codes and delta-time codes. Theevent codes are representative of tones to be produced, and thedelta-time codes are representative of the lapse of time from theinitiation of the performance. A note-on event and a note-off event aretypical examples of the event code. When he or she instructs theautomatic recording/playing system to reproduce the tones, the automaticrecording/playing system starts to sequentially read out the MIDI datacodes from the floppy disc. The automatic recording/playing system movesthe keys without any fingering of a human player, and produces thetones.

However, the picture and performance are respectively recorded in thevideotape cassette and floppy disc, and the video cassette player andautomatic recorder/player system are independent of each other. For thisreason, even if the user concurrently starts the video cassette playerand automatic recording/player system, the synchronization is notguaranteed. The video cassette player reproduces the picture on themonitor screen asynchronously with the reproduction of the tones, and atime lag may take place between the picture and the tones.

Another amateur player wishes to perform a piece of the music on amusical instrument in ensemble with a part of the music reproduced by acompact disc player in the virtual concert hall. The amateur playerprepares a videotape cassette storing video data representative of anorchestral accompaniment and a compact disc storing audio data codesrepresentative of tones of a part of the music. While a video cassetteplayer and a compact disc player are reproducing the picture and theelectronic tones, he or she performs another part of the music on amusical instrument in the virtual concert hall. If the amateur playerwishes to record his or her performance and reproduce it, he or shefingers the piece of music on a musical instrument with an automaticrecording/playing system. The automatic recording/playing system, by wayof example, converts the key actions to MIDI data codes, and stores themin a floppy disc. When he or she instructs the automaticrecording/playing system to reproduce the tones, the automaticrecording/playing system starts to sequentially read out the MIDI datacodes from the floppy disc. The automatic recording/playing system movesthe keys without any fingering of a human player, and produces thetones.

However, the picture, electronic tones and performance are respectivelyrecorded in the videotape cassette, compact disc and floppy disc, andthe video cassette player, compact disc player and automaticrecorder/player system are independent of one another. For this reason,even if the user concurrently starts the video cassette player, compactdisc player and automatic recording/player system, the synchronizationis not guaranteed. The video cassette player and compact disc playerreproduce the picture on the monitor screen and the electronic tonesfrom a sound system asynchronously with the reproduction of the acoustictones, and a tine lug may take place among the picture, electronic tonesand acoustic tones.

SUMMARY OF THE INVENTION

It is therefore an important object of the present invention to providea multimedia platform, which records a performance in synchronizationwith at least video images.

It is also an important object of the present invention to provide arecorder forming a part of the multimedia platform.

It is another important object of the present invention to provide aplayer forming another part of the multimedia platform.

In accordance with one aspect of the present invention, there isprovided a multimedia platform for recording at least first music soundsin an information storage medium synchronously with a picture comprisinga first data source producing a first sort of data containing pieces offirst music data information representative of the first music sounds, asecond data source producing a second sort of data containing pieces ofvideo data information representative of visual images of the pictureand pieces of first time data information representative of a first timedefined from a first viewpoint, a third data source incrementing asecond time defined from the first viewpoint and represented by piecesof second time data information, connected to the second data source soas to compare the second time with the first time to see whether thesecond time is consistent with the first time, modifying the pieces ofsecond time data information with the negative answer so as to eliminatea time difference from between the first time and the second time andconverting the pieces of second time data information to pieces of thirdtime data information representative of a third time defined from asecond viewpoint different from the first viewpoint, a recorderconnected to the first data source and the third data source so as tostore the pieces of first music data information and the pieces of thirdtime data information in the information storage medium, and an imagegenerator connected to the second data source for producing the visualimages.

In accordance with another aspect of the present invention, there isprovided a multimedia platform for reproducing at least first musicsounds synchronously with a picture comprising a first data sourceoutputting a first sort of data containing pieces of first music datainformation representative of the first music sounds and pieces of firsttime data information representative of a first time defined from afirst viewpoint, a second data source outputting a second sort of datacontaining pieces of video data information representative of visualimages of the picture and pieces of second time data informationrepresentative of a second time defined from a second viewpointdifferent from the first viewpoint, an image generator connected to thesecond data source so as to produce the picture from the pieces of videodata information, a sound generator connected for generating the firstmusic sounds from the pieces of first music data information, and atiming controller incrementing a third time defined from the secondviewpoint and represented by pieces of third time data information,connected to the second data source so as to compare the pieces of thirdtime data information with the pieces of second time data information tosee whether or not the third time is consistent with the second time,further connected to the first data source so as to modify the pieces ofsaid first time data information with the negative answer foreliminating a time difference from between the second time and the thirdtime, converting the pieces of first time data information to pieces offourth time data information representative of a fourth time definedfrom said second viewpoint, comparing the pieces of fourth time datainformation with the pieces of third time data information to seewhether or not the third time catches up the fourth time and furtherconnected to the sound generator so as to transfer the pieces of firstmusic data information to the sound generator when the third timecatches up the fourth time.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the multimedia platform, recorder andplayer will be more clearly understood from the following descriptiontaken in conjunction with the accompanying drawings, in which

FIGS. 1A to 1D are block diagrams showing technical concepts ofpreferred embodiments.

FIG. 2A is a block diagram showing the system configuration of amultimedia platform according to the present invention,

FIG. 2B is a view showing an example of a standard MIDI file,

FIG. 3 is a block diagram showing the circuit configuration of acontroller incorporated in a floppy disc controller/driver,

FIG. 4 is a flowchart showing a computer program executed by acorrection value calculator,

FIG. 5 is a timing chart showing a synchronous recording,

FIG. 6 is a block diagram showing the system configuration of acontroller incorporated in another multimedia platform according to thepresent invention,

FIG. 7 is a flowchart showing a computer program executed by anadjuster,

FIG. 8 is a block diagram showing the system configuration of anothermultimedia platform according to the present invention,

FIG. 9 is a block diagram showing the circuit configuration of acontroller incorporated in a disc player forming a part of themultimedia platform,

FIG. 10 is a timing chart showing a synchronous playback,

FIG. 11 is a block diagram showing the system configuration of anothermultimedia platform according to the present invention,

FIG. 12 is a flowchart showing a computer program for controlling thepitch of electronic tones in a synchronous recording,

FIG. 13 is a graph showing the waveform of an electric signalrepresentative of a sound pressure level,

FIG. 14 is a view showing memory locations of a floppy disc for storinga videotape identification code and a standard pitch,

FIG. 15 is a flowchart showing a computer program for a synchronousplayback,

FIGS. 16A to 16C are graphs showing relation between a read-out speedand the pitch of electronic tones,

FIG. 17 is a block diagram showing the system configuration of anothermultimedia platform according to the present invention,

FIG. 18 is a view showing the arrangement of data codes stored in acompact disc,

FIG. 19 is a block diagram showing means incorporated in a dataprocessing unit incorporated in the multimedia platform,

FIG. 20 is a flow chart showing a method for regulating a clock withMIDI time codes,

FIG. 21 is a view showing a standard MIDI file for recording aperformance on a keyboard,

FIG. 22 is a block diagram showing the configuration of a controllerincorporated in a floppy disc recorder,

FIG. 23 is a flowchart showing a sequence of jobs executed by acorrection value calculator,

FIG. 24 is a timing chart showing a synchronous recording carried outthe multimedia platform,

FIG. 25 is a block diagram showing a controller incorporated in a floppydisc recorder of another multimedia platform,

FIG. 26 is a block diagram showing the system configuration of anothermultimedia platform according to the present invention,

FIG. 27 is a block diagram showing the configuration of a controllerincorporated in a floppy disc player,

FIG. 28 is a timing chart showing a synchronous playback carried out themultimedia platform,

FIG. 29 is a block diagram showing another multimedia platform accordingto the present invention,

FIG. 30 is a flowchart showing a method for controlling the pitch ofsecond electronic tones in a synchronous recording,

FIG. 31 is a graph showing the waveform of an electric signalrepresentative of a sound pressure level,

FIG. 32 is a view showing memory locations of a floppy disc for storinga disc identification code and a standard pitch,

FIG. 33 is a flowchart showing a computer program for a synchronousplayback,

FIGS. 34A to 34C are graphs showing relation between a read-out speedand the pitch of electronic tones, and

FIG. 35 is a view showing memory areas of a hard disc unit incorporatedin a modification of the multimedia platform.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention contains four technical concepts shown in FIGS. 1Ato 1D, and the first to eighth embodiments are based on these technicalconcepts.

The first technical concept is illustrated in FIG. 1A. The first andsecond embodiments are based on the first technical concept. Amultimedia platform based on the first technical concept comprises afirst data source 1, a second data source 2, a third data source 4connected to the second data source 2, a recorder connected to the firstand third data sources 1/4 and an image generator 8 connected to thesecond data source.

The first data source 1 produces a first sort of data containing piecesof first music data information representative of first music sounds. Inthe first and second embodiments, an automatic player piano serves asthe first data source 1, and the first music data information and firstmusic sounds are corresponding to music data information stored in MIDIevent codes and acoustic piano tones, respectively. The second datasource 2 produces a second sort of data containing pieces of video datainformation representative of visual images of the picture and pieces offirst time data information representative of a first time defined froma first viewpoint. In the first and second embodiments, a video cameraserves as the second data source 2, and the pieces of video datainformation and the pieces of first time data information are stored invideo data codes and video time codes, respectively. In the first andsecond embodiments, a lapse of time is measured from the firstviewpoint.

The third data source 4 internally increments a second time defined fromthe first viewpoint. The second time is represented by pieces of secondtime data information. The pieces of first time data information areintermittently supplied to the third data source, and the third datasource 4 compare the second time with the first time to see whether thesecond time is consistent with the first time. When the answer is givennegative, the third data source 4 modifies the pieces of second timedata information so as to eliminate a time difference from between thefirst time and the second time. If the answer is given affirmative, thethird data source 4 does not modify the pieces of second time datainformation. Thus, the internal clock incorporated in the third datasource 4 is periodically regulated with the first time. The third datasource 4 is further operative to convert the pieces of second time datainformation to pieces of third time data information representative of athird time defined from a second viewpoint different from said firstviewpoint. In the first and second embodiments, time intervals aredefined from the second viewpoint.

The pieces of first music data information and pieces of third time datainformation are transferred to the recorder 6 so that the recorder 6stores the pieces of first music data information and pieces of thirdtime data information in an information storage medium such as, forexample, a floppy disc.

The pieces of video data information are transferred to the imagegenerator 8 so that the image generator 8 produces the visual images.The multimedia platform based on the first technical concept makes thepieces of first music data information synchronous with the pieces ofvideo data information, because the pieces of third time datainformation are produced from the pieces of second time data informationperiodically modified with the pieces of first time data information.

The second technical concept is illustrated in FIG. 1B, and the thirdand fourth embodiments are based on the second technical concept. Amultimedia platform based on the second technical concept comprises afirst data source 20, a second data source 22, a timing controller 28connected to the first and second data sources 20/22, an image generator24 connected to the second data source 22 and a sound generator 26connected to the timing controller 28. In the third and fourthembodiments, a disc player and a video camera serves as the first datasource 20 and second data source 22, respectively.

The first data source 20 outputs a first sort of data containing piecesof first music data information representative of first music sounds andpieces of first time data information representative of a first timedefined from a first viewpoint. On the other hand, the second datasource 22 outputs a second sort of data containing pieces of video datainformation representative of visual images of a picture and pieces ofsecond time data information representative of a second time definedfrom a second viewpoint different from the first viewpoint. In the thirdand fourth embodiments, time intervals are defined from the firstviewpoint, and a lapse of time is defined from the second viewpoint.

The pieces of video data information are supplied to the image generator24 so that the image generator 24 produces the picture from the piecesof video data information. The sound generator 26 generates the firstmusic sounds from the pieces of first music data informationsynchronously with the picture with the assistance of the timingcontroller 28.

The timing controller 28 internally increments a third time defined fromthe second viewpoint and represented by pieces of third time datainformation. The timing controller 28 compares the pieces of third timedata information with the pieces of second time data information to seewhether or not the third time is consistent with the second time. Whenthe answer is given negative, the timing controller 28 modifies thepieces of said first time data information so as to eliminate a timedifference from between the second time and the third time. Thereafter,the timing controller 28 converts the pieces of first time datainformation to pieces of fourth time data information representative ofa fourth time defined from the second viewpoint, and compares the piecesof fourth time data information with the pieces of third time datainformation to see whether or not the third time catches up the fourthtime. When the answer is given positive, the timing controller 28transfers the pieces of first music data information to the soundgenerator 26, and the sound generator 26 produces the first musicsounds. The pieces of first time data information are modified throughthe comparison between the third time and the second time, and areconverted to the pieces of fourth time data information from the secondviewpoint, which is same as the pieces of second time data information.For this reason, the first music sounds are produced synchronously withthe picture.

The third technical concept is illustrated in FIG. 1C, and the fifth andsixth embodiments are based on the third technical concept. The thirdtechnical concept relates to the first technical concept, and amultimedia platform based on the third technical concept comprises allthe elements of the multimedia platform based on the first technicalconcept, and further comprises a fourth data source 10, a timinggenerator 12 and a sound generator 14. In the fifth and sixthembodiments, a compact disc unit serves as the fourth data source 10,and the sound generator 14 produces second music sounds from pieces ofthird music information synchronously with the picture.

In detail, the fourth data source outputs a third sort of datacontaining the pieces of third music data information representative ofthird music sounds and pieces of fourth time data informationrepresentative of a fourth time defined from the first viewpoint. Thetiming controller 12 internally increments a fifth time defined from thefirst viewpoint and represented by pieces of fifth time datainformation, and compares the pieces of fifth time data information withthe pieces of first time data information to see whether or not thefifth time is consistent with the first time. If the answer is givennegative, the timing controller 12 modifies the pieces of fifth timedata information so as to eliminate a time difference from between thefirst time and the fifth time, and waits for a time at which each of thepieces of second music data information is to be transferred to thesound generator 14. When the fifth time catches up the fourth timerepresented by each of the pieces of fourth time data information, thetiming controller 12 produces an audio signal from the pieces of thirdmusic data information, and supplies the audio signal to the soundgenerator 14. Since the pieces of fifth time data information aremodified with the pieces of first time data information, the soundgenerator 14 generates the second music sounds synchronously with thepicture.

The fourth technical concept is illustrated in FIG. 1D, and the seventhand eighth embodiments are based on the fourth technical concept. Thefourth technical concept relates to the second technical concepts, and amultimedia platform based on the fourth technical concept comprises allthe elements of the multimedia platform based on the second technicalconcept, and further comprises a third data source 30, another timingcontroller 32 and another sound generator 34. In the seventh and eighthembodiments, a compact disc unit serves as the third data source 30. Thethird data source 30, timing controller 32 and sound generator 34 arecorresponding to the fourth data source 10, timing controller 12 andsound generator 14. The third data source 30, timing controller 32 andsound generator 34 makes the music sounds radiated from the soundgenerator 34 synchronous with the picture, and no further description isincorporated hereinafter for avoiding repetition.

First Embodiment

Referring first to FIG. 2A of the drawings, a multimedia platformembodying the present invention is shown and generally indicated at 100.The multimedia platform 100 largely comprises a video camera 102, a discrecorder 104, a sound system 106, a sound source 108, a controller 110and a monitor display 112. The controller 110 is connected to the videocamera 102, disc recorder 104, sound system 106 and sound source 108,and controls these components 102/104/106/108 for recording a piece ofmusic. The video camera 102 is further connected to the monitor display112, and the picture is reproduced on the monitor display 112 during theperformance of a piece of music.

Video Camera

The video camera 102 includes a recorder 114, a player 116 and amanipulating panel 118. A videotape cassette VT is loaded into andunloaded from the video camera 102, and the recorder 114 and player 116are responsive to instructions of a user given through the manipulatingpanel 118 so as to record visual images and sound into and reproducesthem from the videotape cassette VT. In other words, video data codesrepresentative of the visual images and audio data codes representativeof the sound are stored into and read out from the videotape cassetteVT. A picture, i.e., a series of visual images and the sound areproduced from the video data codes on the monitor display 112 and theaudio data codes through the sound system 106.

The recorder 114 includes a video recorder 120, a sound recorder 122 anda time code generator 124. The video recorder 120 has an image pick-updevice (not shown), and the sound recorder 122 is equipped with amicrophone (not shown). When the user wishes to record his or herperformance in the videotape cassette VT, he or she instructs therecorder 114 to record the performance. The image pickup device (notshown) converts the visual images to a video signal, and the videorecorder 120 produces the video data codes from the video signal. On theother hand, the microphone (not shown) converts the sound to an audiosignal, and the sound recorder 122 produces the sound data codes fromthe audio signal. The sound may be tones generated from another musicalinstrument such as, for example, violin performed concurrently with thesound source 108, and the visual images may represent the violinist, whois playing the violin. The time code generator 114 periodically producesa time code representative of the lapse of time after the initiation ofthe recording the visual images. The time code is hereinbelow referredto as “video time code”. RC time codes may be used as the video timecodes. The video data codes are stored in a video track of thevideotape, and the audio data codes are stored in a sound track of thevideotape. The video time codes are also stored in the videotapetogether with the video data codes and sound data codes.

The player 116 is responsive to user's instructions so as to read outthe video data codes and sound data codes from the videotape VT. Theplayer 116 selectively supplies the video data codes and sound datacodes to the monitor display 112 and the controller 110. The monitordisplay 112 is, by way of example, implemented by a liquid crystaldisplay panel. A cathode ray tube is also available for the reproductionof visual images. The monitor display 112 reproduces the picture or theseries of visual images from the video data codes on the screen. Theplayer 116 separately supplies the audio data codes and video time codesto the controller 110.

Controller

The controller 110 includes a code converter 126, a data processing unit128 and a manipulator 130. The code converter 126 is connected betweenthe player 116 and the data processing unit 128, and the player 116 isdirectly connected to the data processing unit 128. A suitable cable maybe used for the connection between the video camera 102 and thecontroller 110. The video time codes are supplied to the code converter126, and the code converter 126 converts the video time codes to musictime codes. The music time codes are hereinbelow referred to as “MIDItime codes”. The MIDI time codes are used for calibrating a clock aswill be described hereinlater, and also represent a lapse of time fromthe initiation of production of the picture and frames. The lapse oftime is defined by hours, minutes and seconds. The MIDI time codes aresupplied from the code converter 126 to the data processing unit 128.The manipulating panel 130, disc recorder 104, sound system 106 andsound source 108 are connected to the data processing unit 128.

User's instructions are given through the manipulating panel 130 to thedata processing unit 128 so that the data processing unit 128 controlsthe video camera 102, disc recorder 104, sound system 106 and soundsource 108 for synchronous recording. One of the main tasks of the dataprocessing unit 126 is to record a performance on the sound source 108synchronously with the playback of a picture. The task is hereinbelowreferred to as “synchronous recording”.

The outline of the synchronous recording is as follows. While the videocamera is reproducing a picture on the monitor display 112, the player116 supplies the video time codes to the code converter 126. The codeconverter 126 converts the video time codes to the MIDI time codes, andsupplies the MIDI time codes to the data processing unit 128. The dataprocessing unit 128 receives the MIDI time codes, and immediatelytransfers the MIDI time codes to the disc recorder 104. The player 116further transfers the audio data codes to the data processing unit 128.The data processing unit 128 produces an audio signal from the audiodata codes, and supplies the audio signal to the sound system 106.Electronic tones are radiated from the sound system 106. The user isassumed to start his or her performance. The sound source 108intermittently produces the event codes representative of the keyactions, and supplies the event codes to the data processing unit 128.The data processing unit 128 immediately transfers the event codes tothe disc recorder 104. The disc recorder 104 produces the delta-timecodes from the MIDI time codes, and stores the event codes anddelta-time codes in an information storage medium such as, for example,a floppy disc FD.

Sound Source

The sound source 108 is broken down into an automatic player piano 132and a tone generator for ensemble 134. The tone generator for ensemble134 is connected to the data processing unit 128, and the event codesare supplied from the data processing unit 128 to the tone generator forensemble 134. The tone generator for ensemble 134 produces digital tonesignal on the basis of the event codes, and converts the digital tonesignal to an analog tone signal. The analog tone signal is supplied fromthe tone generator for ensemble 134 to the sound system 106 so thatelectronic tones are radiated from the sound system 106. If the dataprocessing unit supplies the audio signal to the sound systemconcurrently with the event codes, the parts of the piece of music arereproduced in ensemble.

The automatic player piano 132 includes an acoustic piano 136 and anautomatic playing system 138 and a tone generator for piano tones 140.In this instance, a standard grand piano is used as the acoustic piano136, and includes a keyboard 142, action units 144, hammers 146, strings148, dampers (not shown) and pedals 149. Black keys and white keys arelaid on the well-known pattern, and form parts of the keyboard 142. Theaction units 144 are linked with the black/white keys so that thedepressed keys actuate the associated action units 144. The actuatedaction units 144 drive the associated hammers 146 for free rotation, andthe hammers strike the associated strings 148 at the end of the freerotation for generating acoustic piano tones. The pedals 149 are calledas “damper pedal”, “sustain pedal” and “soft pedal”. When a player stepson the damper pedal, the damper pedal keeps the dampers spaced from thestrings, and the acoustic piano tones are prolonged. The soft pedal isused for lessening the strings struck with the hammers, and the acousticpiano tones are reduced in loudness. The player steps on the sustainpedal after depressing a black/white key or keys. Then, the sustainpedal keeps the associated damper or dampers spaced from the strings,and the acoustic piano tone or tones are prolonged.

The automatic playing system 138 includes a MIDI controller 150,solenoid-operated key actuators 152, key sensors 154, pedal sensors 156and solenoid-operated pedal actuators 158. The MIDI controller 150 isconnected to the data processing unit 128, and event codes are suppliedfrom and to the data processing unit 128. The MIDI controller isresponsive to user's instructions given through the manipulating panel130 for selecting one of the tone generator for piano tones 140 and theautomatic playing system 138.

If the user wishes to generate the electronic tones, he or she instructsthe MIDI controller 150 to transfer the event codes to the tonegenerator for piano tones 140. The MIDI controller 150 transfers theevent codes to the tone generator for piano tones 140, and the tonegenerator for piano tones 140 produces the digital tone signal on thebasis of the event codes, and converts the digital tone signal to theanalog tone signal. The analog tone signal is supplied from the tonegenerator for piano tones 140 to the sound system 106, and electronictones are radiated from the sound system 106.

On the other hand, if the user instructs the MIDI controller 150 toactuate the black/white keys, the MIDI controller 150 determines targettrajectories for plungers of the solenoid-operated key actuators 152.The MIDI controller 150 further determines target trajectories forplungers of the solenoid-operated pedal actuators 149, if necessary. TheMIDI controller 150 selectively supplies driving signals to thesolenoid-operated key actuators 152 and the solenoid-operated pedalactuators 158 so that the plungers project from the solenoid-operatedkey/pedal actuators 152/158 along the target trajectories. The plungersgive rise to the actions of the black/white keys 142 and pedals 149. Forthis reason, the automatic playing system 138 performs a piece of musicwithout any fingering and step of a human player. The key sensors 154and pedal sensors 156 are used in the synchronous recording. While auser is fingering a piece of music on the keyboard 142, the key sensors154 report current key positions to the MIDI controller 150 through keyposition signals, and the pedal sensors 156 report current pedalpositions to the MIDI controller 150. The MIDI controller 150periodically fetches pieces of positional data informationrepresentative of the current key/pedal positions, and analyzes thepieces of positional data information to see whether or not the userdepresses or releases any one of the black/white keys or a pedal 149.When the MIDI controller 150 acknowledges that the black/white keysand/or pedals 149 move, the MIDI controller 150 stores the key actionsuch as a note-on/note off, the note number representative of the pitchof a tone to be reproduced and a velocity representative of the loudnessof the tone and the pedal action in the event codes. The event codes aresupplied from the MIDI controller 150 to the data processing unit 128.

Sound System

The sound system 106 includes a mixer 160, an amplifier 162 and speakers164. The data processing unit 128, tone generator for ensemble 134 andtone generator for piano tones 140 are connected to the mixer 160, andthe audio signal and analog tone signals are selectively supplied to themixer 160. The audio signal and analog tone signals are mixed with oneanother, and the mixed signal is supplied to the amplifier 162. Themixed signal is equalized and amplified by the amplifier 162, and theamplified signal is supplied from the amplifier 162 to the speakers 164.The speakers 164 convert the amplified signal to the electronic tones. Amixer with an input port for digital signals may be used in the soundsystem 106. In this instance, the audio data codes and digital tonesignals are directly supplied to the mixer.

Disc Recorder/Player

The disc recorder/player includes a floppy disc controller/driver 170,and the floppy disc controller/driver 170 has an information processingcapability. The floppy disc controller/driver 170 creates a standardMIDI file in a floppy disc FD during the synchronous recording under thecontrol of the data processing unit 128.

FIG. 2B shows an example of the standard MIDI file SMF. The standardMIDI file SMF is broken down into a header chunk HT and a track chunkTT. Fundamental information such as a chunk type and a videotapeidentification code V-ID are stored in the header chunk HT. Thevideotape identification codes V-ID have been assigned to videotapecassettes, and make each videotape discriminative from the others. Onthe other hand, the track chunk TT is assigned to the MIDI data codesMIDI representative of pieces of music recorded in the floppy disc FD. Aset of MIDI codes MIDI includes event codes and delta time codes. Theevent codes are representative of the tones to be reproduced and thesystem messages such as a system exclusive event, metaevent and soforth. The event codes representative of the tones are produced by theMIDI controller 150, and the event codes representative of the systemmessages are produced by the data processing unit 128. The delta timecodes, which are abbreviated as “ΔT” in FIG. 2B, are representative ofthe time intervals between events and the previous events. The floppydisc controller/driver 170 determines the time intervals between theevents and the previous events, and stores the delta time codes in thetrack chunk together with the event codes.

When a control signal representative of the initiation of synchronousrecording reaches the floppy disc controller/driver 170, the floppy disccontroller/driver 170 starts a clock. The event codes are intermittentlysupplied to the floppy disc controller/driver 170. When an event code orcodes reach the floppy disc controller/driver 170, the floppy disccontroller/driver 170 checks the clock for the arrival time, anddetermines the time interval. In order to record the event codessynchronously with the playback of the picture, the floppy disccontroller/driver 170 compares the lapse of time indicated by the clockwith the lapse of time represented by the MIDI time codes transferredthrough the data processing unit 128 to see whether or not any time lugtakes place. When the answer is given affirmative, the floppy disccontroller/driver 170 varies the time interval in such a manner as toeliminate the time lug from the lapse of time. The floppy disccontroller/driver 170 produces the delta time code representative of thetime interval, and stores the delta time code in the track chunk TTtogether with the event code or codes.

The floppy disc controller/driver 170 includes a controller 172, awrite-in head 174 and a clock generator 176. The controller 172 createsthe standard MIDI file SMF in a floppy disc FD, and writes the codes inthe standard MIDI file SMF through the write-in head 174. The clockgenerator 176 has an oscillator, i.e., the combination of a quartzoscillator and an amplifier and a frequency divider. The oscillatorgenerates a periodic signal, and the frequency divider divides theperiodic signal for producing various clock signals for the timingcontrol. One of the clock signals is called as a tempo clock CT, and thetempo clock CT is supplied to the MIDI controller 150 for generating theMIDI data codes. The time interval between an event and the previousevent is indicated by using the tempo clock CT.

The circuit configuration of the controller 172 is shown in FIG. 3. Thecontroller 172 includes an accumulator 220 serving as the clock, acorrection value calculator 230, a delta-time calculator 240 and a fileproducer 250. The controller 3 is connected to the file producer 250 andthe correction value calculator 230, and supplies the event codes andthe MIDI time codes to the file producer 250 and the correction valuecalculator 230, respectively. The tempo clock CT is supplied from theclock generator 210 to the accumulator 220.

The accumulator 220 includes an adder 221 and a register 222. When thedata processing unit 128 receives the first MIDI time coderepresentative of zero from the code converter 126, the data processingunit 128 writes zero in the register 222. While the controller 110 isrecording the performance synchronously with the picture, the dataprocessing unit 128 transfers the MIDI time code to the correction valuecalculator 230. A source of constant [+1] is connected to one of theinput nodes of the adder 221, and the register 222 is connected to theother input node of the adder 221. The total number N of tempo clocks issupplied to the adder 221, and the adder 221 increments the total numberN of tempo clocks by one. The output node of the adder 221 is connectedto the register 222, and the register 222 is responsive to the tempoclock CT for latching the output signal of the adder 221. Thus, theadder 221 and register 222 form an accumulating loop, and the totalnumber N is incremented by one in response to the tempo clock signal CT.The total number N of tempo clocks is proportional to the lapse of timefrom the reception of the first MIDI time code, i.e., the initiation ofsynchronous recording. Thus, the accumulator serves as the clock.

The file producer 250 is under the control of the data processing unit128. The file producer 250 is connected to the delta-time calculator240, and supplies an instruction signal representative of a calculationof delta time to the delta-time calculator 240 upon reception of anevent code or a set of event codes so that the delta-time calculator 240determines the delta time, i.e., the time interval between the previousevent and the presently received event. The delta-time calculator 240stores the delta-time in a delta-time code, and supplies the delta-timecode to the file producer 250.

The file producer 250 is further connected through a driving circuit(not shown) to the write-in head 174. The data processing unit 128transfers the videotape identification code V-ID to the file producer250, and the file producer 250 writes the videotape identification codeV-ID through the write-in head 260 into the header chunk HT in thefloppy disc FD. While the user is fingering on the keyboard 142, thedata processing unit 128 intermittently transfers the event codes fromthe MIDI controller 150 to the file producer 250. When the event code orcodes reach the file producer 250, the file producer 250 supplies theinstruction signal to the delta-time calculator 240. The delta-timecalculator 240 produces the delta-time code, and supplies it to the fileproducer 250 as described hereinbefore. The file producer 250 writes theevent code or codes, which are supplied from the data processing unit128, and the associated delta-time codes into the track chunk TT of thefloppy disc FD.

The delta-time calculator 240 is connected to the accumulator 220,correction value calculator 230 and file producer 250, and includesregisters 241 and 242. When the control signal representative of theinitiation of synchronous recording reaches the controller 172, theregisters 241/242 are initialized, and zero is written in both registers241 and 242. The time at which the delta-time calculator 240 receivedthe instruction signal from the file producer 250 is stored in theregister 241. The previously instructed time is stored in the register241 as the number Nf of tempo clocks. When the instruction signalreaches the delta-time calculator 240, the delta-time calculator 240reads out the number N of tempo clocks from the register 222, andcalculates the time interval (N−Nf). The delta-time calculator 240 keepsthe number N of tempo clocks in the register 241 as the previousinstructed time Nf. On the other hand, the register 242 is assigned to acorrection value R, which is also written in the form of the number oftempo clocks CT. The correction value R is representative of thedifference between the lapse of time indicated by the clock, i.e., theaccumulator 220 and the lapse of time determined on the basis of theMIDI time code. The correction value R is supplied from the correctionvalue calculator 230, and the delta-time calculator 240 adds thecorrection value R to the time interval (N−Nf) for determining thedelta-time, i.e., (N−Nf+R). The delta-time calculator 240 stores thedelta-time in the delta-time code, and supplies the delta-time code tothe file producer 250. Upon completion of the task instructed by thefile producer 250, the delta-time calculator 240 writes the number N oftempo clocks CK into the register 241. Thus, the previous instructedtime Nf is renewed.

The correction value calculator 230 is connected to the accumulator 220and delta-time calculator 240, and determines the correction value R.The correction value R is representative of the time difference betweenthe lapse of time from the reproduction of the picture and the lapse oftime from the performance on the keyboard 142. The correction valuecalculator 230 determines the correction value R through execution of acomputer program shown in FIG. 4.

A MIDI time code is assumed to reach the correction value calculator230. The correction value calculator 230 starts the computer program atstep SO, and stores the MIDI time code in an internal register (notshown). The MIDI time code represents the lapse of time TCD frominitiation of producing the picture as by step S1.

Subsequently, the correction value calculator 230 reads out the number Nof tempo clocks CT from the register 222, and converts the number N to alapse of time TFD as by step S2. The tempo clocks CT have a pulse periodτ, and the lapse of time TFD is given as (N×τ).

The correction value calculator 240 determines the absolute value of thedifference between the lapse of time TCD and the lapse of time TFD, andcompares the absolute value |TCD−TFD| with a margin Δ to see whether ornot the absolute value |TCD−TFD| is less than the margin Δ as by stepS3.

When the absolute value |TCD−TFD| is less than the margin Δ, the answerat step S3 is given affirmative, and the correction value calculator 230determines that the correction value R is to be zero. Then, thecorrection value calculator 230 writes zero in the register 242 as bystep S4, and exits from the computer program.

On the other hand, the absolute value |TCD−TFD| is greater than themargin Δ, the answer at step S3 is given negative, and the correctionvalue calculator 230 checks the lapses of time TCD and TFD to seewhether the performance on the keyboard 142 is delayed for the pictureas by step S5.

The performance on the keyboard 142 is assumed to be delayed for thepicture. The lapse of time TCD is greater than the lapse of time TFD,and the answer at step S5 is given affirmative. Then, the correctionvalue calculator 230 divides the difference TFD−TCD, which is a negativevalue, by the pulse period τ, and writes the product, i.e., (TCD−TFD)/τin the register 242 as the correction value R. Since the dividend(TCD−TFD) and the divisor τ are a negative value and a positive value,the product (TCD−TFD)/τ is negative. The correction value calculator 230writes the correction value (<0) in the register 242 as by step S6. Whenthe delta-time calculator 240 adds the correction value R to the timeinterval (N−Nf) for determining the delta-time, i.e., (N−Nf+R), the timeinterval (N−Nf) is shortened, and the delta-time code makes the nextnote-on event catches up with the visual images in the picture.

If, on the other hand, the performance on the keyboard 142 is advancedrather than the picture, the answer at step S5 is given negative, andthe correction value calculator 230 divides the difference TFD−TCD,which is a positive value, by the pulse period τ, and writes theproduct, i.e., (TCD−TFD)/τ in the register 242 as the correction valueR. Since the dividend (TCD−TFD) and the divisor τ are positive, theproduct (TCD−TFD)/τ is a positive number. The correction valuecalculator 230 writes the correction value (>0) in the register 242 asby step S7

When the delta-time calculator 240 adds the correction value R to thetime interval (N−Nf) for determining the delta-time, i.e., (N−Nf+R), thetime interval (N−Nf) is prolonged, and the delta-time code makes thevisual images in the picture catch up with the next note-on event.

When the correction value calculator 230 writes the correction value atstep S6 or S7, the correction value calculator 230 terminates the taskat step S8.

Description is hereinafter made on the synchronous recording withreference to FIG. 5. The video time codes, which are read out from thevideotape cassette VT, are converted to the MIDI time codes, which areassigned the first row. The video time codes [0], [0.25], [0.50], . . .are read out at time zero, 0.25 second, 0.50 second . . . , and areimmediately transferred through the data processing unit 128 to thecontroller 172. Thus, the MIDI time codes [k] (k=0, 0.25, 0.50, . . . )are read out at time intervals of 250 milliseconds. In an actualmultimedia platform, the MIDI time codes are produced at time intervalsof 1/30 second. However, the time intervals are reduced to 250milliseconds for the sake of simple description.

The video data codes, which are also read out from the videotapecassette VT, are expressed as p[k], i.e., p[0], p[0.25], p[0.50], . . ., and are read out between time [k] and time [k+1]. The video data codesp[k] are immediately supplied to the monitor display 112 for producing apicture. The second row is assigned to the video data codes p[k].

The audio data codes, which are also read out from the videotapecassette VT, are expressed as a[k] (k=0, 0.25, 0.50, . . . ), and areread out from the videotape between time [k] and time [k+1]. The thirdrow is assigned to the audio data codes a[k]. The audio data codes a[k]are supplied to the data processing unit 128, and are converted to theaudio signal. The fourth row is assigned to the audio data codesconverted to the audio signal.

The fifth row is assigned to the lapse of time r[k], i.e., N×τ, andevent codes ME-1, ME-2, ME-3, . . . are intermittently supplied to thefile producer 250 in response to the fingering on the keyboard 142 asindicated by the sixth row.

A user firstly gives a pause instruction to the data processing unit 128through the manipulating panel 130. The data processing unit 128supplies the control signal representative of the user's instruction tothe floppy disc controller/driver 170 so that the floppy disccontroller/driver 170 enters the idling state. While the data processingunit 128 is waiting for the next instruction, the user loads a floppydisc FD into the floppy disc controller/driver 170 and a videotapecassette VT into the video camera 102. The multimedia platform getsready for the synchronous recording, and informs the user of the readystate through the display window on the manipulating panel 130.

The user instructs the player 116 to start the reproduction of thepicture through the manipulating panel 118. Then, the player 116 readsout the first video time code representative of zero, and supplies thevideo time code to the code converter 126. The code converter 126converts the video time code to the MIDI time code [0], and supplies theMIDI time code [0] to the data processing unit 128. When the MIDI timecode [0] reaches the data processing unit 128, the data processing unit128 supplies the control signal representative of the initiation ofsynchronous recording, i.e., cancellation of the pause instruction tothe controller 172 together with the MIDI time code [0].

With the MIDI time code [0], the registers 222, 241 and 242 are reset tozero, and the accumulator 220 starts to count the tempo clocks CT.Although the correction value calculator 230 gets ready to calculate thelapse of time r[0], the correction value calculator 230 does notcalculate the correction value R on the basis of the MIDI time code [0].

The player 116 further reads out the video data codes p[0] and audiodata codes a[0] from the videotape cassette VT, and supplies the videodata codes p[0] and audio data codes a[0] to the monitor display 112 andthe data processing unit 128, respectively. The monitor display 112starts to produce visual images on the screen, and the data processingunit 128 starts to supply the audio signal to the sound system 106 forradiating the electronic tones from the speakers 164.

When the next video time code is read out from the videotape cassetteVT, the MIDI time code [0.25] is supplied to the correction valuecalculator 230, and the video data codes p[0.25] and audio data codesa[0.25] are transferred to the monitor display 112 and the dataprocessing unit 128. The correction value calculator 230 starts thecomputer program shown in FIG. 4, and stores the correction value R inthe register 242, if necessary. The monitor display 112 continuouslyproduces the visual images on the screen, and the electronic tones areradiated from the speakers 164. The correction value calculator 230calculates the lapse of time r[0.25], and compares the lapse of timer[0.25], i.e., TFD with the lapse of time TCD represented by the MIDItime code [0.25] so see whether or not the correction value R is to bedetermined.

While the MIDI time code is being incremented from [0.25] to [0.75], theplayer 116, data processing unit 128, sound system 106 and thecontroller 172 repeat the above-described jobs, and waits for the firstMIDI event code ME-1. When the user depresses a black/white key, theMIDI controller 150 acknowledges the note-on event, and supplies thefirst MIDI event codes ME-1 through the data processing unit 128 to thefloppy disc controller/driver 170. Upon arrival of the first MIDI eventcodes ME-I at the file producer 250, the file producer 250 requests thedelta-time calculator 240 to determine the lapse of time from theinitiation of the synchronous recording. The delta-time calculator 240reads out the number N of tempo clocks CT from the register 222, andchecks the register 242 for the correction value R. The delta-timecalculator 240 calculates the delta time, i.e., (N−Nf+R), and stores thedelta time in a delta-time code. The delta-time calculator 240 suppliesthe delta-time code to the file producer 250 so that the first MIDIevent codes ME-1 and delta-time code are stored in the track chunk TT bymeans of the write-in head 174.

When the MIDI event codes ME-2/ME-3/ . . . reaches the file producer250, the file producer 250 and delta-time calculator 240 repeat theabove-described jobs for storing the MIDI event codes ME-2/ME-3/ . . .in the track chunk TT together with the delta-time codes.

When the user completes the performance on the keyboard 142, he or shegives the instruction representative of the completion to the dataprocessing unit 128. Then, the data processing unit 128 instructs theplayer 116 to read out the videotape identification code V-ID from thevideotape cassette VT. The player 116 transfers the videotapeidentification code V-ID to the data processing unit 128, and the dataprocessing unit 128 supplies the control signal representative ofstoring the videotape identification code V-ID in the header chunk HT tothe file producer 250 together with the videotape identification codeV-ID. The file producer 250 writes the videotape identification codeV-ID into the header chunk HT, and completes the synchronous recording.

As will be understood from the foregoing description, the correctionvalue calculator 230 periodically regulates the internal clock 220 bythe MIDI time code, and stores the MIDI event codes in the informationstorage medium together with the delta-time codes representative of thetime interval between the MIDI event codes and the previous event codeson the basis of the lapse of time indicated by the clock 220. For thisreason, the performance recorded in the information storage medium isalways synchronized with the picture produced in another informationstorage medium VT.

In the first embodiment, the automatic player piano 132 serves as thefirst data source 1, and the video camera 102 and controller 110 as awhole constitute the second data source 2. The clock generator 176,accumulator 220, delta-time calculator 240 and correction valuecalculator 230 as a whole constitute the third data source 4, and thefile producer 250 and write-in head 174 form in combination the recorder6. The monitor display 112 serve as the image generator 8.

Second Embodiment

FIG. 6 shows another controller 180 incorporated in a floppy disccontroller/driver 181, which in turn is incorporated in anothermultimedia platform embodying the present invention. The other systemcomponents are similar to those of the first embodiment so thatreferences 102/104/106/108/110/112 are used for discriminating them fromone another.

The floppy disc controller/driver 181 also has an information processingcapability. The controller 180 is connected to the data processing unit128. The controller 180 internally produces delta-time codes on thebasis of the number N of tempo clocks CT, and eliminates a timedifference from the lapse of time indicated by the clock upon arrival ofthe MIDI time code. The event codes are supplied from the MIDIcontroller 150 through the data processing unit 128, and the event codesand delta-time codes are written in a floppy disc FD by means of thewrite head 174.

The controller 180 includes an accumulator 220A, a delta-time calculator240A, a file producer 250A and an adjuster 230A. The file producer 250Ais similar to the file producer 250, and no further description ishereinafter incorporated for avoiding repetition.

The accumulator 220A also comprises an adder 221 and a register 222, andincrements the total number N of tempo clocks CT as similar to theaccumulator 220. The total number N expresses the lapse of time from theinitiation of synchronous recording. The difference between theaccumulators 220 and 220A is that the adjuster 230A can rewrite thetotal number N of tempo clocks CT as will be hereinafter described inmore detail.

The delta-time calculator 240A includes only one register 24 1, which isassigned to the total number Nf of the tempo clocks CT at which theprevious event code or codes reached the file producer 250A. Thedelta-time calculator 240A determines a difference between the totalnumber N and the total number Nf, and produces the delta-time coderepresentative of the difference, i.e., the interval between the events.The delta-time calculator 240A supplies the delta-time code to the fileproducer 250A.

When the time code is transferred from the data processing unit 128, theadjuster 230A compares the lapse of time calculated on the basis of thetotal number N with the lapse of time stored in the MIDI time code tosee whether or not the difference between the lapses of time is fallenwithin a predetermined margin Δ. If the difference is equal to or lessthan the margin Δ, the adjuster 230A does not carry out any adjustmentwork. On the other hand, if the difference is greater than the margin Δ,the adjuster 230A rewrites the total number N so as to eliminate thedifference from between the lapses of time.

FIG. 7 illustrates a computer program to be executed by the adjuster230A. A MIDI time code is assumed to reach the adjuster 230A. Theadjuster 230A starts the computer program at step S10, and stores theMIDI time code in an internal register (not shown) as by step S11. TheMIDI time code is representative of the lapse of time TCD frominitiation of producing a picture on the monitor display 112.

Subsequently, the adjuster 230A reads out the total number N of tempoclocks from the register 222, and converts the number N to a lapse oftime TFD from the initiation of synchronous recording as by step S12.The tempo clocks CT have a pulse period τ, and the lapse of time TFD isgiven as (N×τ).

The adjuster 230A determines the absolute value of the differencebetween the lapse of time TCD and the lapse of time TFD, and comparesthe absolute value |TCD−TFD| with the margin Δ to see whether or not theabsolute value |TCD−TFD| is less than the margin Δ as by step S13. Whenthe absolute value |TCD−TFD| is less than the margin Δ, the answer atstep S13 is given affirmative, and the adjuster 736 exits from thecomputer program as by step S14.

On the other hand, the absolute value |TCD−TFD| is greater than themargin Δ, the answer at step S13 is given negative, and the adjuster230A compares the lapse of time TCD with the lapse of time TFD to seewhether or not the internal clock, i.e., accumulator 220A is delayed forthe time stored in the MIDI time code as by step S15.

The internal clock is assumed to be delayed for the lapse of time storedin the time code. The lapse of time TCD is greater than the lapse oftime TFD, and the answer at step S15 is given affirmative. Then, theadjuster 230A divides the absolute value |TFD−TCD| by the pulse periodτ, and add the product, i.e., |TCD−TFD|/τ to the total number N. The sumis written in the register 222 as by step S16. Thus, the internal clockis set with the MIDI time code. The adjuster 736 exits from the computerprogram at step S14.

If, on the other hand, the internal clock is advanced, the answer atstep S15 is given negative, and the adjuster 230A divides the absolutevalue |TCD−TFD| by the pulse period τ, and subtracts the product, i.e.,|TCD−TFD|/τ from the total number N. The adjuster 230A writes thedifference (N−|TCD−TFD|/τ) in the register 222 as by step S17. Thus, theinternal clock is set with the MIDI time code. The adjuster 230A exitsfrom the computer program at step S14.

When a user instructs the controller data processing unit 128 to recordhis or her performance synchronously with a picture stored in avideotape cassette VT, the floppy disc controller/driver 170 internallyproduces the delta-time codes on the basis of the difference between thetotal numbers N and Nf, and stores the event codes and the delta-timecodes in a standard MIDI file SMF. The adjuster 230A periodically checksthe internal clock 220A to see whether or not the lapse of time Nτ isapproximately equal to the lapse of time stored in the MIDI time code.When the lapse of time NT is advanced or delayed, the adjuster 230A setsthe internal clock with the MIDI time code. As a result, the timeinterval stored in the delta-time code is based on the lapse of timeindicated by the MIDI time code, and the performance is recorded in thefloppy disc FD synchronously with the picture on the monitor display112. Thus, the multimedia platform implementing the second embodimentachieves all the advantages of the first embodiment.

In the second embodiment, the clock generator 210, accumulator 220A,adjuster 230A and delta-time calculator 240A as a whole constitute thethird data source 4, and the file producer 250A and write-in head 174form in combination the recorder 6.

Third Embodiment

FIG. 8 shows yet another multimedia platform 300 embodying the presentinvention. The multimedia platform 300 is similar to the multimediaplatform 100 except a disc recorder/player 302 and a controller 304. Forthis reason, the other system components are labeled with referencesdesignating corresponding system components 102/106/108 without detaileddescription for the sake of simplicity.

The disc recorder/player 302 includes the disc recorder 104 and a discplayer 308, and the data processing unit 128 is replaced with a dataprocessing unit 310. The disc recorder 104 is similar to that of thefirst embodiment. The code converter 126 and the MIDI controller 150supply the MIDI time codes and event codes to the disc recorder 104under the control of a data processing unit 310, and the disc recorder104 records a performance on the keyboard 142 in a floppy disc FDsynchronously with the playback of a picture.

The data processing unit 310 achieves other tasks for a synchronousplayback as well as the tasks identical with those of the dataprocessing unit 128, and the disc player 308 supplies the event codes,which are read out from the floppy disc FD, to the data processing unit310 synchronously with the playback of the picture. In detail, a user isassumed to instruct the data processing unit to reproduce theperformance recorded in a floppy disc synchronously with the playback ofa picture through the manipulating panel 130. Then, the data processingunit 310 supplies a control signal to the disc player 308. The controlsignal is representative of the initiation of playback 500 millisecondslater than the initiation of playback of the picture. While the player116 is transferring the video data codes and audio data codes to themonitor display 112 and the data processing unit 310, the video timecodes are periodically supplied to the code converter 126, and areconverted to the MIDI time codes. The data processing unit 310 transfersthe MIDI time codes to the disc player 308 for the synchronizationbetween the picture and the performance.

Prior to description on the disc player 308, the automatic player piano138 is described in more detail. As described in conjunction with themultimedia platform 100, the automatic player piano 132 includes theacoustic piano 136 and automatic playing system 138. Thesolenoid-operated key actuators 152 and solenoid-operated pedalactuators 158 are provided for the keyboard 142 and pedals 149,respectively, and the MIDI controller 150 selectively supplies thedriving signal through the driving circuit 312 to the solenoid-operatedkey/pedal actuators 152/158. The solenoid-operated key actuators 152thus energized with the driving signal give rise to the key motion, andthe hammers 146 are driven for rotation by the associated action units144 so as to strike the associated strings 148 at the end of the freerotation. The strings 148 vibrate, and the acoustic piano tones areradiated from the vibrating strings 148. A time lug takes place betweenthe delivery of the event codes to the MIDI controller 150 and thegeneration of the acoustic piano tones. The time lug is of the order of500 milliseconds in this instance. The time lug of 500 milliseconds isto be taken into account for the synchronous playback. However, in casewhere the user instructs the data processing unit 310 to transfers theevent codes to the tone generator for ensemble 134, any substantialamount of time lug does is not required for the synchronous playback.

The multimedia platform 300 eliminates the time lug from the synchronousplayback as follows. When the user gives the data processing unit 310instructions for the synchronous playback through the manipulating panel130, the data processing unit 310 supplies the control signalrepresentative of the pause instruction to the disc layer 308 and thecontrol signal representative of the initiation of the playback to theplayer 116. When the first MIDI time code reaches the data processingunit 310, the data processing unit 310 gives instructions for theinitiation of playback at a certain point 500 milliseconds after thefirst MIDI code to the disc player 308. The disc player 308 isresponsive to the instructions so as to start the data read-out at thecertain point 500 milliseconds later than the starting point. Althoughthe acoustic piano tone is delayed from the delivery of MIDI codes by500 milliseconds, the disc player 308 reads out the MIDI codes 500milliseconds earlier than the video data codes representative of a scenecorresponding to the acoustic piano tone. Thus, the acoustic tones arereproduced synchronously with the picture.

While the MIDI data codes are being sequentially read out from thefloppy disc FD, the disc player 308 serves as not only a sequencer butalso a timing controller. When the disc player 308 delivers an eventcode or codes to the data processing unit 310, the disc player 308enters waiting state. Upon expiry of the time period indicated by thedelta-time code, the disc player 308 reads out the next event code orcodes from the floppy disc FD, and delivers the read-out event code orcodes to the data processing unit 310. This is the function as thesequencer.

The function as the timing controller is described with reference toFIG. 9. FIG. 9 shows the circuit arrangement of a controller 312incorporated in the disc player 308. The controller 312 includes anevent buffer 314, a delta-time register 316, accumulators 318/320, atransmission control 322 and an adjuster 324 for the function as thetiming controller. The accumulator 318 is implemented by a combinationof an adder 326 and a register 328, and an adder 330 and a register 332constitute the other accumulator 320.

The event code or codes and delta-time code are selectively suppliedfrom the floppy disc FD to the event buffer 314 and delta-time register316, and are stored in the event buffer 314 and the delta-time register316, respectively. A delta-time code may be followed by more than oneevent code. The event buffer 314 has a memory capacity much enough tostore all the event codes concurrently supplied from the floppy disc FD.The value of the delta-time code is equal to the number of tempo clocksCT to be counted between an event and the next event. The event buffer314 is connected to the data processing unit 310, and the delta-timeregister 316 is connected to the accumulator 318 and adjuster 324. Thedelta-time codes are continuously read out from the floppy disc FD until500 milliseconds without any waiting time, and the event codes areignored until 500 milliseconds, if any. For this reason, the accumulatedtotal M is representative of 500 milliseconds immediately after theinitiation of synchronous playback.

The transmission control 322 has two input ports connected to theaccumulator 318 and the adjuster 324, and compare an accumulated totalM, which represents a target time to transfer the event code or codes,with a number N′ stored in the register 332 to see whether or not theevent code or codes are to be transferred to the data processing unit310. When the number N′ reaches the accumulated total M, the answer isgiven affirmative, and the transmission control 322 changes an enablesignal and a latch control signal to an active level, and supplies theactive enable/latch control signals to the data processing unit 310 andthe delta-time register/register for accumulated total 316/328. Thetransmission control 322 may supply the registers 316/328 a write-inclock signal instead of the latch control signal.

The accumulator 318 accumulates the time intervals, i.e., the values ofthe delta-time codes, and supplies the accumulated total M to thetransmission control 322. Each delta-time code is representative of thenumber of tempo clocks CT to be counted between the event and the nextevent so that the accumulated total M is also represented by the totalnumber of tempo clocks CT counted from the initiation of reading out theMIDI codes. The adder 326 has two input ports respectively connected tothe delta-time register 316 and the register for accumulated total 328,and the output port is connected to the register for accumulated total328. Thus, the adder 326 and register 328 form an accumulating loop.When a user instructs the controller 304 to reproduce the performancerecorded in the floppy disc FD, the register 328 is reset to zero. Whilethe disc player 308 is sequentially reading out the MIDI codes, thefloppy disc FD intermittently supplies the delta-time codes to thedelta-time register 316. When the number N′ reaches the accumulatedtotal M, the transmission control 322 changes the latch control signalto the active level. With the active latch control signal, the nextdelta-time code is stored in the delta-time register 316, and isimmediately transferred to the adder 326 for accumulation. The adder 326adds the delta time to the accumulated total M, and the new accumulatedtotal M is stored in the register 328 in the presence of the latchcontrol signal of the active level.

The other accumulator 320 counts the tempo clock CT. The adder 330 hastwo input ports respectively connected to a source of constant value“+1” and the register 332, and the output port of the adder 330 isconnected to the input port of the register 332. The adder 330 andregister 332 form an accumulating loop. The input port, at which theregister 332 is connected to the adder 330, is further connected to theadjuster 324 and the transmission control 322, and the tempo clock CT issupplied to the register 332 as a latch control signal. When the userinstructs the data processing unit 310 to reproduce the performancesynchronously with the picture, an initial value is written in theregister 332. The initial value is equal to 500/τ millisecond. The pulseperiod of the tempo clock CT is represented by τ. The adder 330increments the number by one, and the total is stored in the register332 in response to the tempo clock CT. The number N′ is representativeof the lapse of time from the initiation of the synchronous playback.Thus, the number N′ of the tempo clocks CT is stored in the register332, and is supplied to the adjuster 324 and the transmission control322.

Although the accumulator 318 accumulates the delta-times, the event codeor codes are never transferred to the data processing unit 310 until theaccumulated total M exceeds the number N′ of tempo clocks CT. Afterexceeding the number N′, the tempo clock CT makes the number N′increment. When the number N′ catches up the accumulated total M, theevent code or codes are transferred to the data processing unit 310. Asdescribed hereinbefore, the initial value is “500/τ” so that, even if anevent code or codes are stored in the event buffer 314 before “500/τ”,the event code or codes are not transferred to the data processing unit310.

The adjuster 324 is connected to the data processing unit 310,accumulator 320 and delta-time register 316. The MIDI time codes areperiodically transferred from the code converter 126 through the dataprocessing unit 310 to the adjuster 324, and the accumulator 320supplies the number N′ of tempo clocks CT to the adjuster 324. The lapseof time represented by the MIDI time code is abbreviated as “TCD′”. Theadjuster 324 achieves three major tasks as follows.

The adjuster 324 firstly calculates a lapse of time from the initiationof synchronous playback by multiplying the number N′ by the pulse periodτ of the tempo clocks CT, i.e., (N×τ). As described hereinbefore, theevent codes are transferred to the data processing unit 310 at thecertain point 500 milliseconds later than the initiation of synchronousplayback. In order to equalize the dial plate of one clock to the dialplate of the other clock, the adjuster 241 subtracts 500 millisecondsfrom the lapse of time (N′×τ), and determines a corrected lapse of timeTFD′, i.e., {(N′×τ)−500}. This is the first task.

The second task to be achieved by the adjuster 324 is to set the clockahead or back. First, the adjuster 324 checks the MIDI time code to seewhether or not the lapse of time TCD′ is greater than zero. While theanswer is given negative, the adjuster 324 repeats the comparison. Whena MIDI time code represents the lapse of time greater than zero, theanswer is changed to affirmative. With the positive answer, the adjuster324 compares the lapse of time TFD′ with the lapse of time TCD′ to seewhether the lapse of time TCD′ is greater than, equal to or less thanthe lapse of time TFD′. In case where the lapse of time TFD′ isdifferent from the lapse of time TCD′, the adjuster 324 further checksthe lapses of time TFD′/TCD′ to see whether or not the difference DFtherebetween is fallen within a predetermined margin MG. The adjuster324 proceeds to different steps depending upon the answers as follows.

-   -   Case 1: TFD=TCD or |DF|<MG

The adjuster 324 sets the clock neither ahead nor back. The delta-timecodes are intermittently supplied from the floppy disc FD to thedelta-time register 316, and are accumulated in the register 328. Whenthe total number N′ of the tempo clocks CT reaches the accumulated totalM, the transmission control 322 changes the enable signal and latchcontrol signal to the active level. With the enable signal of the activelevel, the event code or codes are latched in the buffer of the dataprocessing unit 310, and the delta time represented by the nextdelta-time code is accumulated in the accumulator 318.

-   -   Case 2: TCD′>TFD′ and |DF|>MG

The performance reproduced through the automatic player piano 132 isdelayed for the picture reproduced on the monitor display 112 by thedifference DF. The adjuster 324 converts the time lug, i.e., differenceDF to the number DN of tempo clocks CT by dividing the difference DF bythe pulse period τ. The product (TCD′−TFD′)/τ is equivalent to the timedelay. The adjuster 324 takes out the delta-time code from thedelta-time register 316, and subtracts the number DN from the value NDof the delta-time code.

Subsequently, the adjuster 324 checks the calculation result to seewhether or not the difference {ND−(TCD′−TFD′)/τ} is a positive number.When the answer is given affirmative, the adjuster 324 writes thedifference {ND−(TCD′−TFD′)/τ} in the delta-time register 316. The timeinterval represented by the delta-time code is shortened. The adjuster324 supplies the corrected delta-time code to the register 316 so thatthe corrected delta-time code represents the number of tempo clocks CTless than the previous number. When the corrected delta-time code isaccumulated in the register 328, the transmission control 322 transmitsthe event code or codes D3 to the data processing unit 310 earlier thanthe previous schedule. This results in that the delay is canceled. Bothof the performance and picture are synchronously reproduced through theautomatic player piano 312 and the monitor display 112.

On the other hand, if the difference {ND−(TCD′−TFD′)/τ} is a negativenumber, the answer is given negative. In this situation, the adjuster324 divides the product (TCD′−TFD′)/τ by a positive number α, andsubtracts the products (TCD′−TFD′)/τα from the value of the delta-timecode. If the positive number is 2, the difference is given as{ND−(TCD′−TFD′)/2τ}. The adjuster 324 checks the calculation result tosee whether or not the difference is a positive number. When the answeris given affirmative, the adjuster 324 writes the difference{ND−(TCD′−TFD′)/2τ} in the delta-time register 316, and keeps the otherhalf, i.e., (TCD′−TFD′)/2τ in an internal register (not shown). Theadjuster 324 will subtract the other half from the value of the nextdelta time. Thus, the adjuster 324 stepwise takes up the time lug inorder to make the reproduction of performance synchronous with thepicture. If the difference {ND−(TCD′−TFD′)/2τ} is still given negative,the adjuster 324 increases the divisor, and repeats the above-describedsequence.

-   -   Case 3: TFD′>TCD′ and |DF|>MG

In this situation, the performance reproduced through the automaticplayer piano 132 is advanced by the difference DF, i.e., TFD′−TCD′ fromthe reproduction of the picture. The adjuster 324 firstly converts thetime, i.e., difference DF to the number DN of tempo clocks CT bydividing the difference DF by the pulse period τ. The product(TFD′−TCD′)/τ is equivalent to the time by which the performanceproduced by the automatic player piano 132 is advanced. The adjuster 324reads out the delta-time code from the delta-time register 316, and addsthe number DN to the value ND of the delta-time code. The adjuster 324writes the sum {ND+(TFD′−TCD′)/τ} in the delta-time register 316. Thus,the time interval represented by the delta-time code is prolonged. Theadjuster 324 supplies the corrected delta-time code to the register 316so that the corrected delta-time code stored in the register 316represents the number greater than the previous number. When thecorrected delta-time code is accumulated in the register 328, thetransmission control 322 retards the transmission of the event code orcodes. This results in that the picture catches up the performancereproduced through the automatic player piano 132.

FIG. 10 shows a synchronous playback. The MIDI time codes express thelapse of time from the initiation of playback of a picture, and isassigned the first row. [k] is indicative of the lapse of time, and isincremented by 0.25 millisecond Although the video time-codes areusually incremented by 1/30 second, the video time codes shown in FIG.10 is incremented by 0.25 second for the sake of simplicity. Forexample, [0.25] is indicative of the lapse of time 0.25 millisecondsafter the initiation of the playback. The video data codes, which areread out from the videotape cassette VT, are assigned the second row.The video data codes are expressed as “p[k]”. The video data codes p[k]are read out from the videotape cassette VT from time [k] to time [k+1].The video data codes p[0.25] are read out from [0.25] to [0.50].

The audio data codes, which are also read out from the videotapecassette VT, are assigned the third row. The audio data codes areexpressed as “a[k]”. The audio data codes a[k] are read out from thevideotape cassette VT from time [k] to [k+1]. The audio data codes a[k]are supplied to the data processing unit 310, and the data processingunit 310 converts the audio data codes a[k] to the audio signal. Theaudio signal is supplied to the sound system 106, and the electronictones are radiated from the speakers 164. The fourth row is assigned theaudio data codes a[k] converted to the audio signal. Any substantialamount of time delay is not introduced in the conversion from the audiodata codes a[k] to the audio signal so that the audio data codes a[k]converted to the audio signal are put on the vertical lines indicativeof the lapse of time [k] together with the corresponding audio datacodes a[k] read out from the videotape cassette VT.

The MIDI data codes, which are read out from the floppy disc FD, areassigned the fifth row, and are expressed as m[k]. The MIDI data codesm[k] are read out from the floppy disc FD from [k] to [k+1]. Althoughthe players 116 and 308 concurrently start, the MIDI data codes m[k+0.5]are put on the vertical lines together with the corresponding video datacodes p[k] and audio data codes a[k]. The MIDI data codes are brokendown into the event codes and delta-time codes, and the first threeevent codes representative of the note-on are abbreviated as “ME-1”,“ME-2” and “ME-3”. Nevertheless, the delta-time codes between [0] and[0.50] have been already accumulated in the register 328, and theinitial value “500/τ” is written in the register 332 immediately after[0]. The disc player 308 starts to transfer the event codes to the dataprocessing unit 310 at the position 500 milliseconds later than theinitiation of the read-out from the floppy disc FD. For this reason, theMIDI data codes m[0.50] are transferred to the data processing unit 310between [0] and [0.25]

The event codes ME-1, ME-2 and ME-3 are read out from the floppy disc FDat [0.5], [1.00] and [1.50], and are transferred to the data processingunit 310. However, 500 milliseconds are consumed between the delivery tothe MIDI controller 150 and the generation of the acoustic piano tones.For this reason, the acoustic piano tones are generated at [1.00],[1.50] and [2.00] on the basis of the event codes ME-1, ME-2 and ME-3 asshown in the sixth row. The acoustic tone on the basis of the eventcodes ME-1 is generated synchronously with the scene represented by thevideo data codes p[1.00] and electronic tones represented by the audiodata codes a[1.00]. (See the vertical line indicative of [1.00])

A user is assumed to instruct the controller 304 for the synchronousplayback. The data processing unit 310 gives the pause instruction tothe disc player 308 so that the disc player 308 enters the idling state.The player 116 reads out the videotape identification V-ID code from thevideotape cassette VT, and transfers the videotape identification codeV-ID through the data processing unit 310 to the disc player 308. Thedisc player 308 reads out the videotape identification code V-ID fromthe header chunk HT of the standard MIDI file SMF, and compares theread-out videotape identification code V-ID with the videotapeidentification code V-ID supplied from the player 116 to see whether ornot they are consistent with each other. If the videotape identificationcodes V-ID are identical with each other, the disc player 308 reportsthe judgement to the data processing unit 310, and the data processingunit 310 notifies the user of the judgement through the display windowon the manipulating panel 130.

When the user instructs the player 116 to read out the video/audio/videotime codes from the videotape cassette VT through the manipulating panel118, the player starts to read out the video data codes, audio datacodes and video time codes from the videotape cassette VT. The player116 supplies the video time code representative of zero to the codeconverter 126, and the converter 126 supplies the MIDI time code [0] tothe data processing unit 310. The data processing unit 310 supplies thecontrol signal representative of the initiation of synchronous playbackto the disc player 308. When the disc player 308 receives the controlsignal, the disc player 308 resets the register 328 to zero, writes theinitial value “500/τ” into the register 332, and starts to successivelydistribute the event codes and delta-time codes to the event buffer 314and delta-time register 316 without any wait. The delta-time codes aresuccessively accumulated in the register 328 without any wait until theaccumulated total reaches “500/τ”. The disc player 308 immediatelycompletes those jobs so that the MIDI data codes [0.5] are read out fromthe floppy disc FD substantially concurrently with the distribution ofthe video data codes and analog audio signal to the monitor display 112and sound system 106.

The disc player 308 intermittently reads out the event codes anddelta-time codes from the floppy disc FD from m[0.50], and transfers theevent codes through the data processing unit 310 to the MIDI controller150 when the number N′ of tempo clock CT reaches the accumulated totalM. The adjuster 324 periodically corrects the value of the delta-timecodes upon reception of the MIDI time code [k].

The first event code ME-1 representative of the note-on is incorporatedin the MIDI data codes m[1.00], and the MIDI data codes m[1.00] aretransferred to the data processing unit 310 at [0.50]. However, theautomatic player piano 132 consumes 500 milliseconds from the receptionof the event code ME-1 to the generation of the acoustic piano tone. Forthis reason, the acoustic piano tone represented by the event code ME-1is generated at [1.00]. The MIDI data codes m[1.00] are scheduled torealize at [1.00] together with the video data codes p[1.00] and audiodata codes a[1.00]. When the player 116 reads out the video data codesp[1.00] and audio data codes a[1.00], the player immediately transfersthe video data codes p[1.00] and audio data codes a[1.00] to the monitordisplay 112 and the sound system 106, and the monitor display 112 andsound system 106 reproduces the visual images and electronic tones fromthe video data codes p[1.00] and audio data codes a[1.00] at [1.00].Similarly, the event codes ME-2 and ME-3 are incorporated in the MIDIdata codes m[1.50] and m[2.00], and the acoustic tones are generated at[1.50] and [2.00] synchronously with the visual images p[1.50] andp[2.00] and electronic tones a[1.50] and a[2.00]. Thus, the acousticpiano tones are generated synchronously with the picture, i.e., theseries of visual images and electronic tones.

As will be understood from the foregoing description, the multimediaplatform 300 reproduces the acoustic tones synchronously with thepicture and electronic tones.

In the third embodiment, the disc player 308 serves as the first datasource 20, the video camera 102 and controller 304 as a whole constitutethe second data source 22, the monitor display 112 is corresponding tothe image generator 24, the automatic player piano 132 and tonegenerator for ensemble 134 as a whole constitute the sound generator 26,and the controller 312 serves as the timing controller 28.

Fourth Embodiment

FIG. 11 shows still another multimedia platform 350 embodying thepresent invention. The multimedia platform 350 is similar to themultimedia platform 300 except a hard disc unit 352 and a controller354. For this reason, other system components of the multimedia platform350 are labeled with reference numerals designating corresponding systemcomponents of the multimedia platform 300 without detailed descriptionfor the sake of simplicity. The hard disc unit 352 may be replaced withanother sort of memory device such as, for example, a random accessmemory.

The multimedia platform 350 is available for the synchronous recordingand synchronous playback as similar to the multimedia platform 300. Thecontroller 354 adjusts the pitches of the electronic tones to those ofthe corresponding acoustic piano tones through execution of a computerprogram. In detail, the electronic tone produced on the basis of theaudio data code is assumed to have the standard pitch of 443 Hz, i.e.,the pitch of A. If the acoustic piano 136 is tuned to have the standardpitch of 448 Hz, the electronic tones are never harmonized with theacoustic piano tones. In order to make the electronic tones harmonizedwith the acoustic piano tones, the controller 354 controls the pitchesof the electronic tones so that the electronic tones are well harmonizedwith the acoustic piano tones in ensemble.

In order to control the pitches of the electronic tones, the audio datacodes are read out from the videotape cassette VT, and the dataprocessing unit 356 writes the audio data codes in the hard disc unit352. As a result, the audio data codes are read out from the hard discunit 352 independently of the video data codes read out from thevideotape cassette VT.

FIG. 12 shows a computer program for controlling the pitch of theelectronic tones in the synchronous recording. A user is assumed toinstruct the controller 354 on the condition that the pitches of theelectronic tones are to be controlled for harmonization after loading avideotape cassette VT and floppy disc FD into the video camera 102 anddisc recorder 104.

The data processing unit 356 acknowledges the videotape cassette VT andfloppy disc FD loaded into the video camera 102 and disc recorder 104 asby step Sa1, and instructs the player 116 to transfer the audio datacodes from the videotape cassette VT thereto. The data processing unit356 receives the audio data codes, and writes them into the hard discunit 352 as by step Sa2.

Subsequently, the data processing unit 356 checks the manipulating panel130 to see whether or not the user has instructed the pitch control asby step Sa3. If the user has not instructed the data processing unit 356for the pitch control, the answer is given negative “NO”, and the dataprocessing unit 356 proceeds to step Sa9. Jobs at step Sa9 will bedescribed hereinlater. On the other hand, if the user has alreadyinstructed the data processing unit 356 for the pitch control, theanswer is given affirmative “YES”, and the data processing unit 356repeats the loop consisting of steps Sa4, Sa5 and Sa6 for the standardpitch of the electronic tone.

The data processing unit 356 reads out the audio data codes from thehard disc unit 352, and determines the sound pressure level for eachfrequency through a fast Fourier transformation as by step Sa4. FIG. 13shows the waveform of an electric signal representative of soundpressure, which is determined on the basis of the audio data code. Thewaveform has multiple peaks. However, the standard pitch of theelectronic tone is to be close to the standard pitch of the acousticpiano tone. For this reason, the data processing unit may pass theelectric signal through a band pass filter for focusing the analysis onthe target band (440 Hz±α).

Subsequently, the data processing unit 356 selects a certain frequency,and fetches a piece of data information representative of the soundpressure SP at the certain frequency as by step Sa5. The data processingunit 356 compares the sound pressure SP at the certain frequency with athreshold TH to see whether or not the sound pressure at the certainfrequency exceeds the threshold as by step Sa6. If the sound pressure SPis less than the threshold TH, the answer is given negative “NO”. Then,the data processing unit 356 changes the target frequency, and returnsto step Sa4. Thus, the data processing unit 356 changes the targetfrequency, and reiterates the loop consisting of steps Sa4 to Sa6 untilthe answer at step Sa6 is changed to affirmative.

When the data processing unit 356 find the peak P1 (see FIG. 13), theanswer is changed to affirmative “YES”, and the data processing unit 356determines that the certain frequency is the standard pitch as by stepSa7. In the example shown in FIG. 13, the standard pitch is 443 Hz.

Subsequently, the data processing unit 356 acquires the videotapeidentification code V-ID from the video cassette VT through the player116, and informs the disc player 308 of the standard pitch and videotapeidentification code V-ID as by step Sa8. The disc player 308 stores thedata code representative of the standard pitch and videotapeidentification code V-ID in the floppy disc FD as shown in FIG. 14. Incase where the standard MIDI file SMF is to be created in the floppydisc FD, the data code representative of the standard pitch andvideotape identification code V-ID are stored in the header chunk HT.

The data processing unit 356 checks the manipulating panel 130 to seewhether or not the user has instructed the controller 354 for thesynchronous recording as by step Sa9. If the answer at step Sa9 is givennegative “NO”, the data processing unit 356 returns to the main routine.On the other hand, when the answer is given affirmative “YES”, the dataprocessing unit 356 informs the user that the multimedia platform 350gets ready for the synchronous recording, and instructs the discrecorder 104 to record the performance on the keyboard 142 synchronouslywith the picture on the monitor display 112 as by step Sa10. Uponcompletion of the performance, the user instructs the data processingunit 356 to terminate the synchronous recording, and the data processingunit 356 returns to the main routine.

The user is assumed to instruct the controller 354 for reproducing theperformance synchronously with the picture and electronic tones. Thedata processing unit 354 enters a computer program shown in FIG. 15.First, the data processing unit 356 requests the disc player 308 totransfer the event code representative of the standard pitch from thefloppy disc FD thereto as by step Sb1. Subsequently, the data processingunit 356 instructs the manipulating panel 130 to produce a massage suchas “Please depress the white key A” on the display window, and waits forthe user's response. When the user depresses the white key A, the hammer146 strikes the string 148, and the tone A is generated from thevibrating string 148. A microphone (not shown) picks up the tone A, andsupplies the electric signal to the data processing unit 356. The dataprocessing unit analyzes the digital codes, which were converted fromthe electric signal, and determines the standard pitch as by step Sb2.In this instance, the standard pitch for the piano tones is assumed tobe 448 Hz. The data processing unit 356 may measure the sound pressurelevel in a certain band through the fast Fourier transformation, andchecks the sound pressure to see what frequency has the sound pressurelevel over a threshold. When the data processing unit 356 finds thesound pressure level at a certain frequency to exceed the threshold, thedata processing unit 356 determines that the certain frequency is thestandard pitch at the piano tone “A”.

Subsequently, the data processing unit 356 calculates the differencebetween the standard pitch of the electronic tone “A” and the standardpitch of the piano tone “A”, and determines a pitch difference as bystep Sb3. In this instance, the standard pitch of the piano tone “A” is448 Hz, and the standard pitch of the electronic tone “A” is 443 Hz sothat the pitch difference is 5 Hz. The electronic tones are to beincreased in pitch by 5 Hz. Then, the data processing unit 356determines a target speed for reading out the audio data codes from thehard disc unit 352 as by step Sb4. The data read-out speed deeplyconcerns the pitch of tones as follows.

FIGS. 16A to 16C show the relation between the target speed for readingout the audio data codes and the pitch of the electronic tones. Eventhough the audio data codes are not changed, the waveform of the audiosignal representative of the electronic tone is varied depending uponthe target speed for reading out the audio data codes from the hard discunit 352. When the audio data codes are read out from the hard disc unit352 at the standard read-out speed Vb, the audio signal has a waveform Ashown in FIG. 16A. If the data read-out is accelerated, i.e., Vf>Vb, thewaveform A is shrunk, and the audio signal has the waveform B as shownin FIG. 16B. Accordingly, the tone is sharp pitched. The pitch isincreased to 448 Hz. On the other hand, in case where the read-out speedis lowered, i.e., Vs<Vb, the waveform A is expanded, and the audiosignal has the waveform C as shown in FIG. 16C. Accordingly, the pitchof the tone is lowered to 440 Hz.

In this instance, the pitch of the electronic tone is lower than thepitch of the acoustic piano tone by 5 Hz. The data processing unit 356instructs the hard disc unit 352 to increase the data read-out speed.If, on the contrary, the pitch of the second electronic tones is higherthan the pitch of the acoustic piano tones, the data processing unit 356instructs the hard disc unit 352 to decrease the data read-out speed.

When the target speed is determined, the data processing unit 356instructs the hard disc unit 352, player 116 and disc player 308 tostart the synchronous playback under the pitch control as by step Sb5.The player 116 reads out the video data codes and video time codes fromthe videotape cassette VT, the disc player 308 reads out the delta-timecodes and event codes from the floppy disc FD, and the hard disc 352reads out the audio data codes from the hard disc at the target speed asby step Sb6. The video data codes, video time codes and audio data codesare supplied to the monitor display 112, code converter 126 and the dataprocessing unit 356, respectively. The monitor display 112 reproduces apicture on the screen. The video time codes are converted to the MIDItime codes, and the data processing unit 356 transfers the MIDI timecodes to the disc player 308 for controlling the timing to transfer theevent codes through the data processing unit 356 to the MIDI controller150 as similar to those of the third embodiment. The automatic playerpiano 132 produces the acoustic piano tones synchronously with thepicture. The data processing unit 356 converts the audio data codes tothe audio signal, and supplies the audio signal to the sound system 106for producing the electronic tones. Since the audio data codes are readout at the target speed, the electronic tones are well harmonized withthe acoustic piano tones. Thus, the picture, acoustic piano tones andelectronic tones are reproduced as by step Sb7.

Subsequently, the data processing unit 356 checks the videotape cassetteVT, hard disc unit 352 and floppy disc FD to see whether or not all thedata codes have been already read out therefrom as by step Sb8. Whilethe answer at step Sb8 is given negative “NO”, the data processing unit356 returns to step Sb5, and reiterates the loop consisting of steps Sb5to Sb8 until the answer at step Sb8 is changed to affirmative. When thelast video data/video time/audio data/MIDI data codes are read out fromthe videotape cassette VT, hard disc 352 and floppy disc FD, the answerat step Sb8 is changed to affirmative, and the data processing unit 356returns to the main routine.

As will be understood from the foregoing description, the multimediaplatform 350 implementing the fourth embodiment achieves the pitchcontrol between the electronic tones and the acoustic piano tones aswell as the synchronous playback. As a result, the electronic tones arewell harmonized with the acoustic piano tones.

Fifth Embodiment

Turning to FIG. 17, a multimedia platform 400 embodying the presentinvention largely comprises a video camera 402, a floppy disc recorder404, a sound system 406, a sound source 408, a compact disc player 410,a controller 412 and a hard disc unit 414. The multimedia platform 400records MIDI data codes representative of user's performancesynchronously with reproduction of picture and electronic tones on thebasis of audio data codes stored in a videotape cassette VT and furtherwith reproduction of electronic tones on the basis of a compact disc CD.In the following description, the audio data codes stored in thevideotape cassette VT are referred to as “tape-stored audio data codes”,and the electronic tones produced from the tape-stored audio data codesare referred to as “first electronic tones”. On the contrary, the audiodata codes stored in the compact disc CD are referred to as “disc-storedaudio data codes”, and the electronic tones produced from thedisc-stored audio data codes are referred to as “second electronictones”.

The video camera 402, floppy disc recorder 404, sound system 406 andsound source 408 are similar to the video camera 102, disc recorder 104,sound system 106 and sound source 108 incorporated in the multimediaplatform 100. For this reason, the components thereof are labeled withreferences designating corresponding components of the video camera 102,disc recorder 104, sound system 106 and sound source 108 withoutdetailed description for the sake of simplicity.

The compact disc player 410 includes a compact disc controller/driver420, and the controller 412 includes a digital signal processor 422 anda data processing unit 424. The compact disc controller/driver 420 isconnected to the digital signal processor 422 as well as the dataprocessing unit 424. The digital signal processor 422 in turn isconnected to the data processing unit 424 and the mixer 160. The harddisc unit 414 is connected to the data processing unit 424 as similar tothat of the fourth embodiment. The data processing unit 424 supplies acontrol signal to the compact disc controller/driver 420 fortransferring user's instructions, and the compact disc controller/driver420 reads out the disc-stored audio data codes from a compact disc CD.The compact disc controller/driver 420 supplies the disc-stored audiodata codes to the digital signal processor 422, and the digital signalprocessor 422 selectively supplies the disc-stored audio data codes oran analog audio signal to the data processing unit 424 or the mixer 160.

In case where the user instructs the controller 412 to reproduce a pieceof music from the disc-stored audio data codes without thesynchronization with a picture, first electronic tones and acousticpiano tones, the digital signal processor 422 produces the analog audiosignal from the disc-stored audio data codes, and supplies the analogaudio signal to the mixer 160. The analog audio signal is amplified,and, thereafter, is converted to the second electronic tones through thespeakers 164. On the other hand, when the user instructs the controller412 to reproduce a piece of music from the disc-stored audio data codessynchronously with the picture, first electronic tones and acousticpiano tones, the compact disc controller/driver 420 transfers thedisc-stored audio data codes through the digital signal processor 422 tothe data processing unit 424, and the data processing unit 424 writesthe disc-stored audio data codes in the hard disc unit 414 prior to thesynchronous playback. The synchronous playback will be hereinlaterdescribed in detail.

Although compact discs are capable of storing various sorts of datacodes, the disc-stored audio data codes deeply concern the multimediaplatform 400 according to the present invention. In the standard compactdiscs CD for music use, audio data codes ADC1/ADC2 ADC3/ADC4 are storedin the recording area for the right and left channels, and time codesrepresentative of a lapse of time from the initiation of playback areinserted into the audio data codes ADC1/ADC2 ADC3/ADC4 as shown in FIG.18. The standard compact disc CD for the music use further contains atable of contents, and a disc identification code C-ID assigned to thecompact disc CD and music identification codes representative of thetitles of musical compositions. In order to discriminate the time codesstored in the compact disc from the time codes stored in the videotapecassette VT, the audio data codes stored in the compact discs CD and theaudio data codes stored in the videotape cassette VT are hereinafterreferred to as “audio time codes” and “video time codes”, respectively.Another sort of compact discs is shared between the audio data codes andMIDI data codes. The audio data codes and MIDI data codes may be storedin the recording area for the right channel and left channel or viceversa.

The controller 412 controls the other system components402/404/406/408/414 for the synchronous recording and synchronousplayback. In this instance, the floppy disc recorder 404 records aperformance on the keyboard 142 synchronously with reproduction of apicture, first electronic tones and second electronic tones. Prior tothe synchronous recording, the disc-stored audio data codes aretransferred from the compact disc CD to the hard disc unit 414. Whilethe player 116 is reading out the video data codes and video audio datacodes from the videotape cassette VT, the player 116 further reads outthe video time codes from the videotape cassette VT, and transfers thevideo time codes to the code converter 126. The code converter 126converts the video time codes to the MIDI time codes, and supplies theMIDI time codes to the data processing unit 424. The data processingunit 424 supplies the MIDI time codes to the floppy disccontroller/driver 170 so as to produce the delta-time codes without anytime lug between the lapse of time from the initiation of playback andthe lapse of time from the initiation of recording. The data processingunit 424 further controls the data read-out from the hard disc unit 414on the basis of the MIDI time codes, and converts the disc-stored audiodata codes to the analog audio signal. The data processing unit 424transfers event codes from the MIDI controller 150 to the floppy disccontroller/driver 170. Another task assigned to the data processing unit424 is to supply an audio signal converted from the video audio datacodes.

FIG. 19 shows means realized by the data processing unit 424 inconjunction with the data read-out from the hard disc unit 414. Themeans are corresponding to a clock 430, an audio signal generator 432, acorrection value calculator 434 and a clock signal generator 436. Theclock signal generator 436 includes an oscillator implemented by thecombination of a quartz oscillating element and an amplifier (not shown)and a frequency divider (not shown). The oscillator supplies aperiodical signal to the frequency divider, and the frequency divideroutputs clock signals different in frequency from one another. One ofthe clock signals is called as “tempo clocks CT”, and the tempo clocksCT is used for the MIDI data codes. The tempo clocks CT are supplied tothe clock 430, and the clock 430 defines the lapse of time as the numberN of the tempo clocks.

The clock 430 includes an adder 440 and a register 442. The adder 440has two input ports and one output port. The input ports are connectedto a source of constant value [+1] and the output port of the register442, and the output port of the adder 440 is connected to the input portof the register 442. When the user instructs the controller 412 to startthe synchronous playback, the register 442 is reset to zero. Theregister 442 is responsive to the tempo clock CT so as to store thetotal number N of tempo clocks at the output port of the adder 440, andthe adder increments the total number N by one. Thus, the adder 440 andregister 442 form in combination an accumulating loop for accumulatingthe number N of tempo clocks CT after the initiation of the synchronousplayback.

The correction value calculator 434 is connected to the clock 430, andthe MIDI time codes periodically reaches the correction value calculator434. Upon arrival of the MIDI time code, the correction valuecalculator, 434 checks the register 442 to see whether or not any timedifference takes place between the lapse of time stored in the clock 430and the lapse of time indicated by the MIDI time code. If the timedifference is serious, the correction value calculator 434 rewrites thetotal number N for regulating the clock 430 by the MIDI time codes.

FIG. 20 shows a method for regulating the clock 430 by the MIDI timecodes. A MIDI time code is assumed to reach the correction valuecalculator 434. The correction value calculator 434 enters a subroutineprogram at step S20, and stores the MIDI time code in an internalregister (not shown). The MIDI time code represents the lapse of timeTCD from initiation of the synchronous playback as by step S21.

Subsequently, the correction value calculator 434 reads out the number Nof tempo clocks CT from the register 442, and converts the number N to alapse of time TFD as by step S22. The tempo clocks CT have a pulseperiod τ, and the lapse of time TFD is given as (N×τ).

The correction value calculator 434 determines the absolute value of thedifference between the lapse of time TCD and the lapse of time TFD, andcompares the absolute value |TCD−TFD| with a margin Δ to see whether ornot the absolute value |TCD−TFD| is less than the margin Δ as by stepS23. When the absolute value |TCD−TFD| is less than the margin Δ, theanswer at step S23 is given affirmative “YES”, and the correction valuecalculator 434 return to the main routine program.

On the other hand, the absolute value |TCD−TFD| is greater than themargin Δ, the answer at step S23 is given negative “NO”, and thecorrection value calculator 434 checks the lapses of time TCD and TFD tosee whether the clock 430 is delayed for the MIDI time code as by stepS24.

The clock 430 is assumed to be delayed for the MIDI time code. The lapseof time TCD is greater than the lapse of time TFD, and the answer atstep S24 is given affirmative “YES”. Then, the correction valuecalculator 434 divides the difference TCD−TFD by the pulse period τ, anddetermines the correction value (TCD−TFD)/τ. The correction valuecalculator 434 adds the correction value (TCD−TFD)/τ to the total numberN stored in the register 442 as by step S25. Thus, the correction valuecalculator 434 sets ahead the clock 430.

If, on the other hand, the clock 430 is advanced rather than the MIDItime code, the answer at step S24 is given negative “NO”, and thecorrection value calculator 434 divides the difference TCD−TFD by thepulse period τ, and determines the correction value (TCD−TFD)/τ. Thecorrection value calculator 434 subtracts the correction value(TCD−TFD)/τ from the total number N as by step S26 so that thecorrection value calculator 434 sets the clock 430 back. Upon completionof the job at step S25 or S26, the correction value calculator 434returns to the main routine program.

Turning back to FIG. 19, the audio signal generator 432 controls thedata read-out form the hard disc unit 414, and is connected to the clock430, hard disc unit 414 and mixer 160. The audio signal generator 432includes a data read-out controller 444 and a signal generator 446. Thedata read-out controller 444 sequentially reads out the disc-storedaudio data codes and audio time codes from the hard disc unit 414.

When the disc-stored audio data codes are transferred from the datareadout controller 444 to the signal generator 446, the data read-outcontroller 444 reads out the next audio time code, and periodicallyfetches the total number N from the register 442. The data read-outcontroller 444 multiplies the total number by the pulse period τ. Theproduct Nτ is representative of the lapse of time. Then, the dataread-out controller 444 compares the lapse of time Nτ with the lapse oftime indicated by the audio time code to see whether or not thedisc-stored audio data codes are to be read out. When the lapse of timeNτ catches up the lapse of time indicated by the audio time code, thedata read-out controller 444 transfers the next disc-stored audio datacodes to the signal generator 446.

The signal generator 446 converts the disc-stored audio data codes tothe analog audio signal, and supplies the analog audio signal to themixer 414 for generating the second electronic tones. Since thecorrection value calculator 434 periodically regulates the clock 430with the MIDI time codes, the lapse of time Nτ is also consistent withthe lapse of time indicated by the MIDI time codes. The audio signalgenerator 432 produces the analog audio signal from the disc-storedaudio data codes at the time when the lapse of time Nτ catches up theassociated audio time code. For this reason, the second electronic tonesare produced synchronously with the picture produced on the monitordisplay 112.

The floppy disc recorder 404 is operative to record a performance on thekeyboard 142 synchronously with reproduction of a picture and firstelectronic tones and/or reproduction of the second electronic tones. Thefloppy disc recorder 404 creates a standard MIDI file SMF in a floppydisc FD under the control of the data processing unit 424. FIG. 21 showsthe standard MIDI file SMF to be created in the floppy disc FD. Thestandard MIDI file SMF1 has a header chunk HT1 and a track chunk TT1. Inthis instance, a disc identification code C-ID assigned to the compactdisc CD is stored in the header chunk HT1 together with the fundamentalinformation such as the chunk type and videotape identification codeV-ID. Although only the videotape identification code V-ID is stored inthe header chunk HT (see FIG. 2B), the information stored in the headerchunk HT1 makes the compact disc CD and videotape cassette VT, fromwhich the picture and a piece of music were produced during therecording, discriminative from other compact discs and other videotapecassette. On the other hand, event codes and delta-time codes ΔT arestored in the track chunk TTI. Each delta-time code ΔT is insertedbetween an event code or codes and the previous event code or codes, andis representative of the time period therebetween. Most of the eventcodes represents the fingering on the keyboard, and other event codesrepresent a system exclusive event, metaevent and so forth. The eventcodes and delta-time codes AT form a set of MIDI data codesrepresentative of a piece of music performed on the keyboard 142. Amusic identification code representative of the title of the compositionmay be further stored in the header chunk HT1.

When the user instructs the controller 412 to record a performancesynchronously with a picture reproduced from the video data codes and apiece of music reproduced from the disc-stored audio data codes, thefloppy disc recorder 404 starts an internal clock, and periodicallyregulates the clock with the MIDI time codes. The MIDI controller 150supplies the event codes representative of the depressed keys, releasedkeys, velocity and so fourth to the data processing unit 424, and thedata processing unit 424 transfers the event codes to the floppy discrecorder 404. Upon arrival of the event code or codes, the floppy discrecorder 404 determines the time period between the event code or codesand the previous event code or codes, and writes the event code or codesand the delta-time code representative of the time period in the trackchunk TT1.

FIG. 22 shows a controller 450 and a write-in head 452 both incorporatedin the floppy disc recorder 404. The clock signal generator 436 suppliesthe tempo clock CT to the controller 450. The controller 450 includes anaccumulator 454 serving as the clock, a correction value calculator 456,a delta-time calculator 458 and a file producer 460. The controller 412is connected to the file producer 460 and the correction valuecalculator 456, and supplies the event codes and the MIDI time codes tothe file producer 460 and the correction value calculator 456,respectively. The tempo clock CT is supplied from the clock signalgenerator 436 to the accumulator 454.

The accumulator 454 includes an adder 462 and a register 464. When thedata processing unit 424 receives the first MIDI time coderepresentative of zero from the code converter 126, the data processingunit 424 writes zero in the register 464. While the controller 450 isrecording a performance synchronously with the picture and secondelectronic tones, the data processing unit 424 transfers the MIDI timecode to the correction value calculator 456. A source of constant [+1]is connected to one of the input ports of the adder 462, and theregister 464 is connected to the other input port of the adder 462. Thetotal number N of tempo clocks is supplied to the adder 462, and theadder 462 increments the total number N of tempo clocks by one. Theoutput port of the adder 462 is connected to the input port of theregister 464, and the register 464 is responsive to the tempo clock CTfor latching the output signal of the adder 462. Thus, the adder 462 andregister 464 form an accumulating loop, and the total number N isincremented by one in response to the tempo clock signal CT. The totalnumber N of tempo clocks is proportional to the lapse of time from thereception of the first MIDI time code, i.e., the initiation ofsynchronous recording. Thus, the accumulator serves as the clock.

The file producer 460 is under the control of the data processing unit424. The file producer 424 is connected to the delta-time calculator458, and supplies an instruction signal representative of a calculationof delta time to the delta-time calculator 458 upon reception of anevent code or event codes so that the delta-time calculator 458determines the delta time, i.e., the time interval between the previousevent and the presently received event. The delta-time calculator 458stores the delta-time in the delta-time code, and supplies thedelta-time code to the file producer 460.

The file producer 460 is further connected through a driving circuit(not shown) to the write-in head 452. The data processing unit 424transfers the videotape identification code V-ID and disc identificationcode C-ID to the file producer 460, and the file producer 460 writes thevideotape identification code V-ID and disc identification code C-IDthrough the write-in head 452 into the header chunk HT1 in the floppydisc FD. While the user is fingering on the keyboard 142, the dataprocessing unit 424 intermittently transfers the event codes from theMIDI controller 150 to the file producer 460. When the event code orcodes reach the file producer 460, the file producer 460 supplies theinstruction signal to the delta-time calculator 458. The delta-timecalculator 458 produces the delta-time code, and supplies it to the fileproducer 460 as described hereinbefore. The file producer 460 writes theevent code or codes, which have been supplied from the data processingunit 424, and the associated delta-time codes into the track chunk TT ofthe floppy disc FD.

The delta-time calculator 458 is connected to the accumulator 454,correction value calculator 456 and file producer 460, and includesregisters 466 and 468. When the control signal representative of theinitiation of synchronous recording reaches the controller 450, theregisters 466/468 are initialized, and zero is written in both registers466 and 468. The time at which the delta-time calculator 456 receivedthe instruction signal from the file producer 460 is stored in theregister 464. The previously instructed time is stored in the register466 as the number Nf of tempo clocks. When the instruction signalreaches the delta-time calculator 458, the delta-time calculator 458reads out the number N of tempo clocks from the register 464, andcalculates the time interval (N−Nf). On the other hand, the register 468is assigned to a correction value R, which is also written in the formof the number of tempo clocks CT. The correction value R isrepresentative of the difference between the lapse of time indicated bythe clock, i.e., the accumulator 454 and the lapse of time determined onthe basis of the MIDI time code. The correction value R is supplied fromthe correction value calculator 456, and the delta-time calculator 458adds the correction value R to the time interval (N−Nf) for determiningthe delta-time, i.e., (N−Nf+R). The delta-time calculator 458 stores thedelta-time in the delta-time code, and supplies the delta-time code tothe file producer 460. Upon completion of the task instructed by thefile producer 460, the delta-time calculator 458 writes the number N oftempo clocks CK into the register 468 as the previous value Nf. Thus,the previous instructed time Nf is renewed.

The correction value calculator 456 is connected to the accumulator 454and delta-time calculator 458, and determines the correction value R.The correction value R is representative of the time difference betweenthe lapse of time from the reproduction of the picture and the lapse oftime from the performance on the keyboard 142. The correction valuecalculator 456 determines the correction value R through execution of acomputer program shown in FIG. 23.

A MIDI time code is assumed to reach the correction value calculator456. The correction value calculator 456 starts the computer program atstep S30, and stores the MIDI time code in an internal register (notshown). The MIDI time code represents the lapse of time TCD frominitiation of producing the picture as by step S31.

Subsequently, the correction value calculator 456 reads out the number Nof tempo clocks CT from the register 464, and converts the number N to alapse of time TFD from the reception of the first MIDI time coderepresentative of zero as by step S32. The tempo clocks CT have a pulseperiod τ, and the lapse of time TFD is given as (N×τ).

The correction value calculator 456 determines the absolute value of thedifference between the lapse of time TCD and the lapse of time TFD, andcompares the absolute value |TCD−TFD| with a margin Δ to see whether ornot the absolute value |TCD−TFD| is less than the margin Δ as by stepS33. When the absolute value |TCD−TFD| is less than the margin Δ, theanswer at step S33 is given affirmative “YES”, and the correction valuecalculator 456 determines that the correction value R is to be zero.Then, the correction value calculator 456 writes zero in the register468 as by step S34, and returns to the main routine program.

On the other hand, the absolute value |TCD−TFD| is greater than themargin Δ, the answer at step S33 is given negative “NO”, and thecorrection value calculator 456 checks the lapses of time TCD and TFD tosee whether or not the clock 454 is delayed for the time indicated bythe MIDI time code as by step S35.

The clock is assumed to be delayed for the time indicated by the MIDItime code. The lapse of time TCD is greater than the lapse of time TFD,and the answer at step S35 is given affirmative “YES”. Then, thecorrection value calculator 456 divides the difference TFD−TCD, which isa negative value, by the pulse period τ, and writes the product, i.e.,(TCD−TFD)/τ in the register 468 as the correction value R. Since thedividend (TCD−TFD) and the divisor τ are a negative value and a positivevalue, the product (TCD−TFD)/τ is negative. The correction valuecalculator 456 writes the correction value (<0) in the register 468 asby step S36. When the delta-time calculator 456 adds the correctionvalue R to the time interval (N−Nf) for determining the delta-time,i.e., (N−Nf+R), the time interval (N−Nf) is shortened, and thedelta-time code makes the next note-on event catches up with thecorresponding scene.

If, on the other hand, the clock is advanced rather than the timeindicated by the MIDI time code, the answer at step S35 is givennegative, and the correction value calculator 456 divides the differenceTFD−TCD, which is a positive value, by the pulse period τ, and writesthe product, i.e., (TCD−TFD)/τ in the register 468 as the correctionvalue R. Since the dividend (TCD−TFD) and the divisor τ are positive,the product (TCD−TFD)/τ is a positive number. The correction valuecalculator 456 writes the correction value R (>0) in the register 468 asby step S37.

When the delta-time calculator 458 adds the correction value R to thetime interval (N−Nf) for determining the delta-time, i.e., (N−Nf+R), thetime interval (N−Nf) is prolonged, and the delta-time code makes thescene catch up with the next note-on event.

When the correction value calculator 456 writes the correction value atstep S36 or S37, the correction value calculator 456 terminates the taskat step S38.

FIG. 24 shows the synchronous recording. The video time codes, which areread out from the videotape cassette VT, are converted to the MIDI timecodes, which are assigned the first row. The video time codes [0],[0.25], [0.50], . . . are read out at time zero, 0.25 second, 0.50second, . . . , and are immediately transferred through the dataprocessing unit 424 to the controller 450. Thus, the MIDI time codes [k](k=0, 0.25, 0.50, . . . ) are read out at time intervals of 250milliseconds. In an actual multimedia platform, the MIDI time codes areproduced at time intervals of 1/30 second. However, the time intervalsare reduced to 250 milliseconds for the sake of simple description.

The video data codes, which are also read out from the videotapecassette VT, are expressed as p[k], i.e., p[0], p[0.25], p[0.50], . . ., and the video data codes p[k] are read out between time [k] and time[k+1]. The video data codes p[k] are immediately supplied to the monitordisplay 112 for producing a picture. The second row is assigned to thevideo data codes p[k].

The tape-stored audio data codes, which are also read out from thevideotape cassette VT, are expressed as va[k] (k=0, 0.25, 0.50, . . . ),and the tape-stored audio data codes va[k] are read out from thevideotape between time [k] and time [k+1]. The third row is assigned tothe tape-stored audio data codes va[k]. The tape-stored audio data codesva[k] are supplied to the data processing unit 424, and are converted tothe analog audio signal. The fourth row is assigned to the tape-storedaudio data codes converted to the analog audio signal.

The fifth row is assigned to the disc-stored audio data codes ca[k],which are read out from the hard disc unit 414 between time [k] and time[k+1]. The disc-stored audio data codes ca[k] are supplied to the dataprocessing unit 424, and the data processing unit 424 converts thedisc-stored audio data codes ca[k] to the analog audio signal.

The sixth row is assigned to the lapse of time r[k], i.e., N×τ, andevent codes ME-1, ME-2, ME-3, . . . are intermittently supplied to thefile producer 460 in response to the fingering on the keyboard 142 asindicated by the seventh row.

A user firstly gives instructions for synchronous recording to the dataprocessing unit 424 through the manipulating panel 130. The dataprocessing unit 128 supplies the control signal representative ofselecting a certain piece of music and, thereafter, transferring thedisc-stored audio data codes ca[k] to the hard disc unit 414 to thecompact disc controller/driver 420, and the control signalrepresentative of pause to the floppy disc controller/driver 170. Thefloppy disc controller/driver 170 enters the idling state. The compactdisc controller/driver 420 selects the certain piece of music from thecompact disc CD, and transfers the disc-stored audio data codes ca[k]through the digital signal processor 422 to the data processing unit424. The data processing unit 424 writes the disc-stored audio datacodes ca[k] into the hard disc unit 414.

Upon completion of the data transfer, the multimedia platform 400 getsready for the synchronous recording, and informs the user of the readystate through the display window on the manipulating panel 130.

The user instructs the player 116 to start the reproduction of thepicture and first electronic tones through the manipulating panel 118.The player 116 reads out the first video time code representative ofzero, and supplies the video time code to the code converter 126. Thecode converter 126 converts the video time code to the MIDI time code[0], and supplies the MIDI time code [0] to the data processing unit424. When the MIDI time code [0] reaches the data processing unit 424,the data processing unit 424 supplies the control signal representativeof the initiation of synchronous recording, i.e., cancellation of thepause instruction to the controllers 424/450 together with the MIDI timecode [0].

With the MIDI time code [0], the registers 442, 462, 464 and 468 arereset to zero, and the clock 430 and accumulator 454 start to count thetempo clocks CT. Although the correction value calculators 434/456 getready to calculate the lapse of time, the correction value calculators434/456 do not calculate the correction value R on the basis of the MIDItime code [0].

The player 116 further reads out the video data codes p[0] andtape-stored audio data codes va[0] from the videotape cassette VT, andsupplies the video data codes p[0] and tape-stored audio data codesva[0] to the monitor display 112 and the data processing unit 424,respectively. Similarly, the audio signal generator 432 reads out thedisc-stored audio data codes ca[0] from the hard disc unit 414 of thedata processing unit 424. The monitor display 112 starts to producevisual images on the screen. The data processing unit 424 starts toproduce the analog audio signals from the tape-stored audio data codesand disc-stored audio data codes, and supplies both analog audio signalsto the sound system 406 for radiating the first electronic tones andsecond electronic tones from the speakers 164.

When the next video time code is read out from the videotape cassetteVT, the MIDI time code [0.25] is supplied to the correction valuecalculators 434/456, and the video data codes p[0.25]/tape-stored audiodata codes va[0.25] and the disc-stored audio data codes ca[0.25] aretransferred to the monitor display 112 and the data processing unit 424.The monitor display 112 continuously produces the visual images on thescreen, and the data processing unit 424 converts the tape-stored audiodata codes va[0.25] to the analog audio signal so as to generate thefirst electronic tones.

The correction value calculators 434/456 start to execute the computerprograms shown in FIGS. 20 and 23 at [0.25]. If the calculation valuecalculators 434/456 find the individual time differences to be larger invalue than the given margins A, the correction value calculator 434changes the number N of the accumulated tempo clocks CT, and thecorrection value calculator 456 stores the correction values R in theregister 468. The controller 424 regulates the clock 430 with the MIDItime codes upon arrival of each MIDT time code, and the controller 450determines the correction value R also upon arrival of each MIDI timecode.

When the audio time code reaches the audio signal generator 432 from thehard disc unit 414 after [0.25], the audio signal generator 432 waitsfor the time at which the lapse of time No catches up the lapse of timeindicated by the audio time code, and converts the disc-stored audiodata signal ca[k] to the analog audio signal for generating the secondelectronic tones. Thus, the picture, first electronic tones and secondelectronic tones are reproduced synchronously with one another.

While the MIDI time code is being incremented from [0.25] to [0.75], theplayer 116, data processing unit 424, sound system 406, hard disc unit414 and the controller 450 repeat the above-described jobs, and waitsfor the first MIDI event code ME-1. When the user depresses ablack/white key, the MIDI controller 150 acknowledges the note-on event,and supplies the first MIDI event codes ME-1 through the data processingunit 424 to the floppy disc controller/driver 170. Upon arrival of thefirst MIDI event codes ME-1 at the file producer 460, the file producer460 requests the delta-time calculator 458 to determine the lapse oftime from the initiation of the synchronous recording. The delta-timecalculator 458 reads out the number N of tempo clocks CT from theregister 464, and checks the register 468 for the correction value R.The delta-time calculator 458 calculates the delta time, i.e., (N−Nf+R),and stores the delta time in a delta-time code. The delta-timecalculator 458 supplies the delta-time code to the file producer 460 sothat the first MIDI event codes ME-1 and delta-time code are stored inthe track chunk TT1 by means of the write-in head 452.

When the MIDI event codes ME-2/ME-3/ . . . reach the file producer 460,the file producer 460 and delta-time calculator 458 repeat theabove-described jobs for storing the MIDI event codes ME-2/ME-3/ . . .in the track chunk TT1together with the delta-time codes.

When the user completes the performance on the keyboard 142, he or shegives the instruction for the completion of synchronous playback to thedata processing unit 424. Then, the data processing unit 424 instructsthe player 116 and compact disc converter/driver 420 to read out thevideotape identification code V-ID and disc identification code C-IDfrom the videotape cassette VT and compact disc CD, respectively. Theplayer 116 and compact disc converter/driver 420 transfer the videotapeidentification code V-ID and disc identification code C-ID to the dataprocessing unit 424, respectively, and the data processing unit 424supplies the control signal representative of storing the videotapeidentification code V-ID and disc identification code CID in the headerchunk HT1 to the file producer 460 together with the videotapeidentification code V-ID and disc identification code C-ID. The fileproducer 460 writes the videotape identification code V-ID and discidentification code C-ID into the header chunk HT1, and completes thesynchronous recording.

As will be understood from the foregoing description, the correctionvalue calculators 434 periodically regulates the internal clock 430 withthe MIDI time codes, and the correction value calculator 456periodically determines the correction value R through the comparisonbetween the accumulator 454 and the MIDI time codes. The audio signalgenerator 432 converts the discstored audio data codes to the analogaudio signal at the time when the clock 430 catches up the audio timecodes. The delta-time calculator 458 takes the correction value R intoaccount, and produces the delta-time codes. Thus, the multimediaplatform 400 reproduces the second electronic tones synchronously withthe picture and first electronic tones, and records the performance inthe floppy disc FD synchronously with the picture, first electronictones and second electronic tones.

In the fifth embodiment, the compact disc unit 410 and digital signalprocessor 422 as a whole constitute the fourth data source 10, and theclock 430, audio signal generator 432, correction value calculator 434and clock generator 436 as a whole constitute timing generator 12. Thesound system 106 serves as the sound generator 14.

Sixth Embodiment

FIG. 25 shows another controller 500 incorporated in a floppy disccontroller/driver 502, which in turn is incorporated in anothermultimedia platform embodying the present invention. The other systemcomponents are similar to those of the fifth embodiment so thatreferences 402/406/408/410/412/414 are used in the following descriptionfor discriminating them from one another.

The floppy disc controller/driver 502 also has an information processingcapability. The controller 500 is connected to the data processing unit412. The controller 500 internally produces delta-time codes on thebasis of the number N of tempo clocks CT, and eliminates a timedifference from the lapse of time indicated by the clock upon arrival ofthe MIDI time codes. The event codes are supplied from the MIDIcontroller 150 through the data processing unit 412, and the event codesand delta-time codes are written in a floppy disc FD by means of thewrite head 452.

The controller 500 includes an accumulator 454A, a delta-time calculator458A, a file producer 460A and an adjuster 456A. The file producer 460Ais similar to the file producer 460, and no further description ishereinafter incorporated for avoiding repetition.

The accumulator 454A also comprises an adder 461A and a register 464A,and increments the total number N of tempo clocks CT as similar to theaccumulator 454. The total number N expresses the lapse of time from theinitiation of synchronous recording. The difference between theaccumulators 454 and 454A is that the adjuster 456A can rewrite thetotal number N of tempo clocks CT as will be hereinafter described inmore detail.

The delta-time calculator 458A includes only one register 468A which isassigned to the total number Nf of the tempo clocks CT at which theprevious event code or codes reached the file producer 460A. Thedelta-time calculator 458A determines a difference between the totalnumber N and the total number Nf, and produces the delta-time coderepresentative of the difference, i.e., the interval between the events.The delta-time calculator 458A supplies the delta-time code to the fileproducer 460A.

When the time code is transferred from the data processing unit 424, theadjuster 456A compares the lapse of time NT with the lapse of timeindicated by the MIDI time code to see whether or not the differencebetween the lapses of time is fallen within a predetermined margin Δ. Ifthe difference is equal to or less than the margin Δ, the adjuster 456Adoes not carry out any regulation. On the other hand, if the differenceis greater than the margin Δ, the adjuster 456A rewrites the totalnumber N so as to eliminate the difference from between the lapses oftime.

The other system components behave as similar to those of the multimediaplatform 400. For this reason, no further description is incorporatedhereinafter for the sake of simplicity. The multimedia platformimplementing the sixth embodiment achieves all the advantages of thefifth embodiment.

Seventh Embodiment

FIG. 26 shows another multimedia platform 600 embodying the presentinvention. The multimedia platform 600 is similar to the multimediaplatform 400 except for a floppy disc recorder/player 602 and acontroller 604. The other system components are similar to those of themultimedia platform 400. For this reason, the other system componentsare labeled with the references designating corresponding systemcomponents of the multimedia platform 400 without detailed description.The multimedia platform 600 records a performance on the keyboard 142and reproduces second electronic tones from discstored audio data codessynchronously with reproduction of a picture and first electronic tonesas similar to the multimedia platform 400. The multimedia platform 600is further operative to reproduce the acoustic piano tones on the basisof the event codes and the second electronic tones from the disc-storedaudio data codes synchronously with the reproduction of the picture andfirst electronic tones. This operation is hereinafter referred to as“synchronous playback”.

The floppy disc recorder/player 602 includes a floppy disc player 606 aswell as the floppy disc recorder 404, and are connected to a dataprocessing unit 608 incorporated in the controller 604. The dataprocessing unit 608 is responsive to user's instruction representativeof the synchronous playback so that the data processing unit 608controls the video camera 402, compact disc unit 410, floppy disc player606 and automatic player piano 132 for reproducing a picture, acousticpiano tones and first and second electronic tones. The data processingunit 608 instructs the floppy disc player 606 to start to transfer theevent codes to the data processing unit 608 500 from the certainposition equivalent to 500 milliseconds later than the head of the trackchunk TTI. This is because of the fact that a time lug takes placebetween the delivery of the event codes to the MIDI controller 150 andthe generation of the acoustic piano tones. In this instance, the timelug is 500 milliseconds. The data read-out from the floppy disc FD isadvanced rather than the reproduction of the picture and first andsecond electronic tones by 500 milliseconds. For this reason, the timelug is cancelled.

If, on the other hand, the user instructs the data processing unit 608to transfer the event codes to the tone generator for ensemble 134. Thetime lug is ignoreable. The data processing unit 608 instructs thefloppy disc player 606 to read out the MIDI data code from the head ofthe track chunk TT1.

Assuming now that the user instructs the controller 604 to start thesynchronous playback through the automatic player piano 132, the dataprocessing unit 608 gives the pause instruction to the floppy discplayer 606, and instructs the compact disc unit 410 to read out thedisc-stored audio data codes from the compact disc CD. The disc-storedaudio data codes are transferred to the data processing unit 608, andthe data processing unit 608 stores the disc-stored audio data codes inthe hard disc unit 414.

Upon completion of the data transfer to the hard disc unit 414, the dataprocessing unit 608 notifies the user of the completion of the datatransfer to the hard disc unit 414. When the user gives the instructionfor the synchronous playback to the player 116 through the manipulatingpanel 118, the player 116 starts to supply the video data codes,tape-stored audio data codes and video time codes to the monitor display112, data processing unit 608 and code converter 126, respectively. Themonitor display 112 reproduces the picture from the video data codes,and the data processing unit 608 produces the analog audio signal fromthe tape-stored audio data codes. The code converter 126 converts thevideo time codes to the MIDI time codes, and supplies the MIDI timecodes to the data processing unit 608. The data processing unit 608controls the conversion from the disc-stored audio data codes to theanalog audio signal with the MIDI time codes, and supplies the MIDI timecodes to the floppy disc player 606 for controlling the transfer ofevent codes.

Especially, when the MIDI time codes representative of zero reaches thefloppy disc player 606, the floppy disc player 606 continuously readsout the MIDI data codes from the track chunk TTI until the floppy discplayer 606 receives the delta-time code read out from the position 500milliseconds later than the head of the track chunk TT1 without any datatransfer of event codes to the data processing unit 608. The MIDI datacodes may be stored in the floppy disc FD by means of the floppy discrecorder 404 as similar to the fifth embodiment. The floppy disc player606 starts to transfer the event codes stored from the positionequivalent to 500 milliseconds later than the head to the dataprocessing unit 608. The continuous data read-out from the head of thetrack chunk is immediately completed so that the floppy disc player 606starts the transmission of event codes substantially concurrently withthe data read-out from the videotape cassette VT and hard disc unit 414.

The floppy disc player 606 serves as a sequencer and a timingcontroller. When a delta-time code reaches the floppy disc player 606,the floppy disc player 606 enters the idling state for the time periodindicated by the delta-time code, and restarts the data read-out uponexpiry of the time period. This is the function of the sequencer. Thefunction of the timing controller is hereinafter described withreference to FIGS. 27 and 28.

FIG. 27 shows the circuit arrangement of a controller 612 incorporatedin the floppy disc player 606. The controller 612 includes an eventbuffer 614, a delta-time register 616, accumulators 618/620, atransmission control 622 and an adjuster 624. The accumulator 618 isimplemented by a combination of an adder 626 and a register 628, and anadder 630 and a register 632 constitute the other accumulator 620.

The event code or codes and delta-time code are selectively suppliedfrom the floppy disc FD to the event buffer 614 and delta-time register616, and are stored in the event buffer 614 and the delta-time register616, respectively. A delta-time code may be followed by more than oneevent code. The event buffer 614 has a memory capacity much enough tostore all the event codes concurrently supplied from the floppy disc FD.The value of the delta-time code is equal to a number of tempo clocks CTto be counted between an event and the next event. The event buffer 614is connected to the data processing unit 608, and the delta-timeregister 616 is connected to the accumulator 618 and adjuster 624. Thedelta-time codes are continuously read out from the floppy disc FD untilthe position equivalent to 500 milliseconds without any waiting time,and the event codes are ignored until the position equivalent to 500milliseconds, if any. For this reason, the accumulated total M isrepresentative of 500 milliseconds immediately after the initiation ofsynchronous playback.

The transmission control 6322 has two input ports connected to theaccumulator 618 and the adjuster 624, and compare an accumulated totalM, which represents a target time to transfer the event code or codes,with a number N′ stored in the register 632 to see whether or not theevent code or codes are to be transferred to the data processing unit608. When the number N′ reaches the accumulated total M, the answer isgiven affirmative, and the transmission control 622 changes an enablesignal and a latch control signal to an active level, and supplies theactive enable/latch control signals to the data processing unit 608 andthe delta-time register/register for accumulated total 616/628. Thetransmission control 622 may supply the registers 616/628 a writeinclock signal instead of the latch control signal.

The accumulator 618 accumulates the time intervals, i.e., the values ofthe delta-time codes, and supplies the accumulated total M to thetransmission control 622. Each delta-time code is representative of anumber of tempo clocks CT to be counted between the event and the nextevent so that the accumulated total M is also represented by the totalnumber of tempo clocks CT counted from the initiation of reading out theMIDI codes. The adder 626 has two input ports respectively connected tothe delta-time register 616 and the register for accumulated total 628,and the output port is connected to the register for accumulated total628. Thus, the adder 626 and register 628 form an accumulating loop.When a user instructs the controller 604 to reproduce the performancerecorded in the floppy disc FD, the register 628 is reset to zero. Whilethe floppy disc player 606 is sequentially reading out the MIDI codes,the floppy disc FD intermittently supplies the delta-time codes to thedelta-time register 616. When the number N′ reaches the accumulatedtotal M, the transmission control 622 changes the latch control signalto the active level. With the active latch control signal, the nextdelta-time code is stored in the delta-time register 616, and isimmediately transferred to the adder 626 for accumulation. The adder 626adds the delta time to the accumulated total M, and the new accumulatedtotal M is stored in the register 628 in the presence of the latchcontrol signal of the active level.

The other accumulator 620 counts the tempo clock CT. The adder 630 hastwo input ports respectively connected to a source of constant value“+1” and the register 632, and the output port of the adder 630 isconnected to the input port of the register 632. The adder 630 andregister 632 form an accumulating loop. The input port, at which theregister 632 is connected to the adder 630, is further connected to theadjuster 624 and the transmission control 622, and the tempo clock CT issupplied to the register 632 as a latch control signal. When the userinstructs the data processing unit 608 to reproduce the performance andthe second electronic tones synchronously with the picture and firstelectronic tones, an initial value is written in the register 632. Theinitial value is equal to 500/τ millisecond. The pulse period of thetempo clock CT is represented by τ. The adder 630 increments the numberby one, and the total is stored in the register 632 in response to thetempo clock CT. The number N′ is representative of the lapse of timefrom the reception of the MIDI time code representative of zero or theinitiation of synchronous playback. Thus, the number N′ of the tempoclocks CT is stored in the register 632, and is supplied to the adjuster624 and the transmission control 622.

Although the accumulator 618 accumulates the delta-times, the event codeor codes are never transferred to the data processing unit 608 until theaccumulated total M exceeds the number N′ of tempo clocks CT. Afterexceeding the number N′, the tempo clock CT makes the number N′increment. When the number N′ catches up the accumulated total M, theevent code or codes are transferred to the data processing unit 608. Asdescribed hereinbefore, the initial value is “500/τ” so that, even if anevent code or codes are stored in the event buffer 614 before “500/τ”,the event code or codes are not transferred to the data processing unit608.

The adjuster 624 is connected to the data processing unit 608,accumulator 620 and delta-time register 616. The MIDI time codes areperiodically transferred from the code converter 126 through the dataprocessing unit 608 to the adjuster 624, and the accumulator 620supplies the number N′ of tempo clocks CT to the adjuster 624. The lapseof time represented by the MIDI time code is abbreviated as “TCD′”. Theadjuster 624 achieves three major tasks as follows.

The adjuster 624 firstly calculates a lapse of time from the initiationof synchronous playback by multiplying the number N′ by the pulse periodτ of the tempo clocks CT, i.e., (N×τ). As described hereinbefore, theevent codes are transferred to the data processing unit 608 at thecertain point 500 milliseconds later than the initiation of thesynchronous playback. In order to equalize the dial plate of one clockto the dial plate of the other clock, the adjuster 624 subtracts 500milliseconds from the lapse of time (N′×τ), and determines a correctedlapse of time TFD′, i.e., {(N′×τ)−500}. This is the first task.

The second task to be achieved by the adjuster 624 is to set the clockahead or back. First, the adjuster 624 checks the MIDI time code to seewhether or not the lapse of time TCD′ is greater than zero. While theanswer is given negative, the adjuster 624 repeats the comparison. Whena MIDI time code represents the lapse of time greater than zero, theanswer is changed to affirmative. With the positive answer, the adjuster624 compares the lapse of time TFD′ with the lapse of time TCD′ to seewhether the lapse of time TCD′ is greater than, equal to or less thanthe lapse of time TFD′. In case where the lapse of time TFD′ isdifferent from the lapse of time TCD′, the adjuster 624 further checksthe lapses of time TFD′/TCD′ to see whether or not the difference DFtherebetween is fallen within a predetermined margin MG. The adjuster624 proceeds to different steps depending upon the answers as follows.

-   -   Case 1: TFD=TCD or |DF|<MG

The adjuster 624 sets the clock neither ahead nor back. The delta-timecodes are intermittently supplied from the floppy disc FD to thedelta-time register 616, and are accumulated in the register 628. Whenthe total number N′ of the tempo clocks CT reaches the accumulated totalM, the transmission control 622 changes the enable signal and latchcontrol signal to the active level. With the enable signal of the activelevel, the event code or codes are latched in the buffer of the dataprocessing unit 608, and the delta time represented by the nextdelta-time code is accumulated in the accumulator 618.

-   -   Case 2: TCD′>TFD′ and |DF|>MG

The performance reproduced through the automatic player piano l 32 isdelayed for the picture reproduced on the monitor display 112 by thedifference DF. The adjuster 624 converts the time lug, i.e., differenceDF to the number DN of tempo clocks CT by dividing the difference DF bythe pulse period τ. The product (TCD−TFD)/τ is equivalent to the timedelay. The adjuster 624 takes out the delta-time code from thedelta-time register 616, and subtracts the number DN from the value NDof the delta-time code.

Subsequently, the adjuster 624 checks the calculation result to seewhether or not the difference {ND−(TCD′−TFD′)/τ} is a positive number.When the answer is given affirmative, the adjuster 624 writes thedifference {ND(TCD′−TFD′)/τ} in the delta-time register 616. The timeinterval represented by the delta-time code is shortened. The adjuster624 supplies the corrected delta-time code to the register 616 so thatthe corrected delta-time code represents the number of tempo clocks CTless than the previous number. When the corrected delta-time code isaccumulated in the register 628, the transmission control 622 transmitsthe event code or codes to the data processing unit 608 earlier than theprevious schedule. This results in that the delay is canceled. All ofthe performance, picture and first and second electronic tones aresynchronously reproduced through the automatic player piano 312, monitordisplay 112 and the sound system 406.

On the other hand, if the difference {ND−(TCD′−TFD′)/,} is a negativenumber, the answer is given negative. In this situation, the adjuster624 divides the product (TCD′−TFD′)/τ by a positive number α, andsubtracts the products (TCD′-TFD′)/τα from the value of the delta-timecode. If the positive number is 2, the difference is given as{ND−(TCD′−TFD′)/2τ}. The adjuster 624 checks the calculation result tosee whether or not the difference is a positive number. When the answeris given affirmative, the adjuster 624 writes the difference{ND−(TCD′−TFD′)/2τ} in the delta-time register 616, and keeps the otherhalf, i.e., (TCD′−TFD′)/2τ in an internal register (not shown). Theadjuster 624 will subtract the other half from the value of the nextdelta time. Thus, the adjuster 624 stepwise takes up the time lug inorder to make the reproduction of performance synchronous with thepicture. If the difference {ND−(TCD′−TFD′)/2τ} is still given negative,the adjuster 624 increases the divisor, and repeats the above-describedsequence.

-   -   Case 3: TCD′<TFD′ and |DF|>MG

In this situation, the performance reproduced through the automaticplayer piano 132 is advanced by the difference DF, i.e., TFD′−TCD′ fromthe reproduction of the picture. The adjuster 624 firstly converts thetime, i.e., difference DF to the number DN of tempo clocks CT bydividing the difference DF by the pulse period τ. The product(TFD′−TCD′)/τ is equivalent to the time by which the performanceproduced by the automatic player piano 132 is advanced. The adjuster 624reads out the delta-time code from the delta-time register 616, and addsthe number DN to the value ND of the delta-time code. The adjuster 624writes the sum {ND+(TFD′−TCD′)/τ} in the delta-time register 616. Thus,the time interval represented by the delta-time code is prolonged. Theadjuster 624 supplies the corrected delta-time code to the register 616so that the corrected delta-time code stored in the register 616represents the number greater than the previous number. When thecorrected delta-time code is accumulated in the register 628, thetransmission control 622 retards the transmission of the event code orcodes. This results in that the picture catches up the performancereproduced through the automatic player piano 132.

FIG. 28 shows a synchronous playback. The MIDI time codes express thelapse of time from the initiation of playback of a picture, and isassigned the first row. [k] is indicative of the lapse of time, and isincremented by 0.25 millisecond. Although the video time codes areusually incremented by 1/30 second, the video time codes shown in FIG.28 is incremented by 0.25 second for the sake of simplicity. Forexample, [0.25] is indicative of the time 0.25 milliseconds later thanthe initiation of the playback.

The video data codes, which are read out from the videotape cassette VT,are assigned the second row. The video data codes are expressed as“p[k]”. The video data codes p[k] are read out from the videotapecassette VT from time [k] to time [k+1]. For example, the video datacodes p[0.25] are read out from [0.25] to [0.50].

The tape-stored audio data codes, which are also read out from thevideotape cassette VT, are assigned the third row. The audio data codesare expressed as “va[k]”. The audio data codes va[k] are read out fromthe video-tape cassette VT from time [k] to [k+1]. The audio data codesva[k] are supplied to the data processing unit 608, and the dataprocessing unit 608 immediately converts the audio data codes va[k] tothe analog audio signal. The audio signal is supplied to the soundsystem 106, and the first electronic tones are radiated from thespeakers 164. The fourth row is assigned the tapestored audio data codesva[k] converted to the audio signal. Any substantial amount of timedelay is not introduced in the conversion from the tape-stored audiodata codes va[k] to the analog audio signal so that the tape-storedaudio data codes va[k] converted to the analog audio signal are put onthe vertical lines indicative of the lapse of time [k] together with thecorresponding audio data codes va[k] read out from the videotapecassette VT.

The disc-stored audio data codes are assigned the fifth row. Thediscstored audio data codes are expressed as “ca[k]”. The disc-storedaudio data codes ca[k] are read out from the hard disc 414 from time [k]to [k+1]. The data processing unit 608 immediately converts thedisc-stored audio data codes ca[k] to the analog audio signal. The audiosignal is supplied to the sound system 106, and the second electronictones are radiated from the speakers 164. Since the data processing unit608 periodically regulates the clock 430 with the MIDI time codes, thedisc-stored audio data codes ca[k] are converted to the analog audiosignal concurrently with the tape-stored audio data codes va[k], and thedisc-stored audio data codes ca[k] converted to the analog audio signalare put on the vertical lines indicative of the lapse of time [k]together with the corresponding tape-stored audio data codes va[k]converted to the analog audio signal.

The MIDI data codes, which are read out from the floppy disc FD, areassigned the sixth row, and are expressed as m[k]. The MIDI data codesm[k] are read out from the floppy disc FD from [k] to [k+1]. Althoughthe players 116 and 606 concurrently start, the MIDI data codes m[0] tom[0.25] are continuously read out from the floppy disc FD without anyinterval for accumulating the delta-times in the register 628. For thisreason, the MIDI data codes m[k+0.5] are put on the vertical linestogether with the corresponding video data codes p[k], tape-stored audiodata codes va[k] and disc-stored audio data codes ca[k]. The MIDI datacodes are broken down into the event codes and delta-time codes, and thefirst three event codes representative of the note-on are abbreviated as“ME-1”, “ME-2” and “ME-3”.

The event codes ME-1, ME-2 and ME-3 are read out from the floppy disc FDat [0.5], [1.00] and [1.50], and are transferred to the data processingunit 608. However, 500 milliseconds are consumed between the delivery tothe MIDI controller 150 and the generation of the acoustic piano tones.For this reason, the acoustic piano tones are generated at [1.00],[1.50] and [2.00] on the basis of the event codes ME-1, ME-2 and ME-3 asshown in the seventh row.

A user is assumed to give instructions to carry out the synchronousplayback to the controller 604. The data processing unit 608 gives thepause instruction to the floppy disc player 606 so that the disc player606 enters the idling state. The data processing unit 608 instructs thecompact disc unit 410 to read out and transfer the disc-stored audiodata codes, and the data processing unit 608 writes the disc-storedaudio data codes in the hard disc unit 414. Upon completion of the datawrite-in, the data processing unit 608 notifies the user of the read forstart.

The user instructs the player 116 to read out the video data codes,tape-stored audio data codes and video time codes at [0]. The player 116starts to read out the video/audio/video time codes from the videotapecassette VT. The player 116 supplies the video time code representativeof zero to the code converter 126, and the converter 126 supplies theMIDI time code [0] to the data processing unit 608. With the MIDI timecode [0], the data processing unit 608 resets the clock 430, and startsto read out the disc-stored audio data codes ca[0] from the hard disc414. The correction value calculator 434 periodically regulates thenumber of tempo clocks CT in the register 442 with the MIDI time code[k] from [0.25]. For this reason, the disc-stored audio data codes ca[k]are converted to the analog audio signal synchronously with theconversion from the tape-stored audio data codes va[k] to the analogaudio signal.

The data processing unit 608 transfers the MIDI time code [0] to thefloppy disc player 606. When the floppy disc player 606 receives theMIDI time code [0], the floppy disc player 606 resets the register 628to zero, writes the initial value “500/τ” into the register 632, andstarts to selectively distribute the event codes and delta-time codes tothe event buffer 614 and delta-time register 616 without any wait. Thedelta-time codes are continuously accumulated in the register 628without any wait until the accumulated total reaches “500/τ”. The discplayer 606 immediately completes those jobs so that the MIDI data codes[0.5] are read out from the floppy disc FD substantially concurrentlywith the distribution of the video/analog audio signals to the monitordisplay/sound system 112/106.

The floppy disc player 606 intermittently reads out the event codes anddelta-time codes from the floppy disc FD from m[0.50], and transfers theevent codes through the data processing unit 608 to the MIDI controller150 when the number N′ of tempo clock CT reaches the accumulated totalM. The event codes are delivered to the MIDI controller 150 500milliseconds before the read-out of the video data codes p[k]. Theadjuster 624 periodically corrects the value of the delta-time codesupon reception of the MIDI time codes [0.25], [0.50] . . . so that thepicture, first and second electronic tones and acoustic piano tones aresynchronously reproduced from p[0.75], va[0.75], ca[0.75] and m[0.75].

The first event code ME-1 representative of the note-on is incorporatedin the MIDI data codes m[1.00], and the MIDI data codes m[1.00] aretransferred to the data processing unit 608 at [0.50]. However, theautomatic player piano 132 consumes 500 milliseconds from the receptionof the event code ME-1 to the generation of the acoustic piano tone. Forthis reason, the acoustic piano tone represented by the event code ME-1is generated at [1.00]. The MIDI data codes m[1.00] are scheduled torealize at [1.00] together with the video data codes p[1.00],tape-stored audio data codes va[1.00] and discstored audio data codesca[1.00]. When the player 116 and data read out control 444 read out thevideo data codes p[1.00]/tape-stored audio data codes va[1.00] anddisc-stored audio data codes ca[1.00], the player 606, data processingunit 608 and signal generator 446 immediately transfer the video datacodes p[1.00], analog audio signal corresponding to the tape-storedaudio data codes va[1.00] and analog audio signal corresponding to thedis-stored audio data codes ca[1.001] to the monitor display 112 and thesound system 106, and the monitor display 112 and sound system 106reproduce the visual images and first and second electronic tones at[1.00]. Thus, the synchronous playback is achieved.

Similarly, the event codes ME-2 and ME-3 are incorporated in the MIDIdata codes m[1.50] and m[2.00], and the acoustic tones are generated at[1.50] and [2.00] synchronously with the visual images p[1.50] andp[2.00] and first and second electronic tones va[1.50]/ca[1.50] andva[2.00] ca[2.00]. Thus, the acoustic piano tones are generatedsynchronously with the picture, i.e., the series of visual images, firstelectronic tones and second electronic tones.

As will be understood from the foregoing description, the multimediaplatform 300 reproduces the acoustic tones and second electronic tonessynchronously with the picture and first electronic tones.

In the synchronous playback described in conjunction with FIG. 28, thedata processing unit 608 may instruct the player 116 and floppy discpayer 606 to read out and transfer the tape identification codes V-ID.The data processing unit 608 compares the tape identification codes V-IDto see whether or not they are consistent with each other. If the answeris given negative, the data processing unit 608 notifies the user of theinconsistency, and waits for the next user's instruction. On the otherhand, when the answer is given affirmative, the data processing unit 608instructs the player 116 to start to read out the video/tape-storedaudio/video time codes from the videotape cassette VT. This feature isdesirable, because the performance is always reproduced together withthe corresponding picture.

The data processing unit 608 may check the disc identification codesC-ID to see whether or not they are consistent with each other, or checkboth of the tape identification codes and disc-identification codesbefore the synchronous playback.

In the seventh embodiment, the compact disc unit 410 and digital signalprocessor 422 as a whole constitute the third data source 30, and theclock 430, audio signal generator 432, correction value calculator 434and clock signal generator 436 as a whole constitute the timingcontroller 32. The sound system 106 serves as both sound generators26/34.

Eighth Embodiment

FIG. 29 shows another multimedia platform 700 embodying the presentinvention. The multimedia platform 700 is similar to the multimediaplatform 600 except a controller 702. For this reason, other systemcomponents of the multimedia platform 700 are labeled with referencenumerals designating corresponding system components of the multimediaplatform 600 without detailed description for the sake of simplicity.

The multimedia platform 700 is available for the synchronous recordingand synchronous playback as similar to the multimedia platform 600. Thecontroller 702 adjusts the pitches of the second electronic tones tothose of the corresponding acoustic piano tones through execution of acomputer program. In detail, the second electronic tone produced on thebasis of the disc-stored audio data code is assumed to have the standardpitch of 443 Hz, i.e., the pitch. If the acoustic piano 132 is tuned tohave the standard pitch of 448 Hz, the electronic tones are neverharmonized with the acoustic piano tones. In order to make the secondelectronic tones well harmonized with the acoustic piano tones, thecontroller 702 controls the pitches of the second electronic tones sothat the second electronic tones are well harmonized with the acousticpiano tones in ensemble.

In order to control the pitches of the electronic tones, the disc-storedaudio data codes are read out from a compact disc CD, and the dataprocessing unit 704 writes the disc-stored audio data codes in the harddisc unit 414 before the synchronous playback. The controller 702achieves all the tasks assigned to the controller 604, and furtherachieves the following tasks.

FIG. 30 shows a computer program for controlling the pitch of the secondelectronic tones in the synchronous recording. A user is assumed toinstruct the controller 702 on the condition that the pitches of thesecond electronic tones are controlled for harmonization after loading afloppy disc FD and compact disc CD into the floppy disc recorder 404 andcompact disc unit 410.

The data processing unit 704 acknowledges the compact disc CD and floppydisc FD loaded into the compact disc unit 410 and floppy disc recorder404 as by step Sc1, and instructs the compact disc unit 410 to read outand transfer the disc-stored audio data codes from the compact disc CD.The data processing unit 704 receives the disc-stored audio data codestransferred through the digital signal processor 422, and writes theminto the hard disc unit 414 as by step Sc2.

Subsequently, the data processing unit 704 checks the manipulating panel130 to see whether or not the user has instructed the pitch control asby step Sc3. If the user has not instructed the data processing unit 704to control the disc-stored audio data codes for the harmonization withthe acoustic piano tones, the answer is given negative “NO”, and thedata processing unit 704 proceeds to step Sc9. Jobs at step Sc9 will bedescribed hereinlater. On the other hand, if the user has alreadyinstructed the data processing unit 704 to control the disc-stored audiodata codes for the harmonization, the answer at step Sc3 is givenaffirmative “YES”, and the data processing unit 704 repeats the loopconsisting of steps Sc4, Sc5 and Sc6 for determining the standard pitchof the electronic tone.

The data processing unit 704 reads out the disc-stored audio data codesfrom the hard disc unit 414, and determines the sound pressure level foreach frequency through a fast Fourier transformation as by step Sc4.FIG. 31 shows the waveform of an electric signal representative of soundpressure, which is determined on the basis of the disc-stored audio datacode. The waveform has multiple peaks. However, the standard pitch ofthe electronic tone is to be close to the standard pitch of the acousticpiano tone. For this reason, the data processing unit may pass theelectric signal through a band pass filter for focusing the analysis onthe target band (440 Hz±α).

Subsequently, the data processing unit 704 selects a certain frequency,and fetches a piece of data information representative of the soundpressure SP at the certain frequency as by step Sc5. The data processingunit 704 compares the sound pressure SP at the certain frequency with athreshold TH to see whether or not the sound pressure at the certainfrequency exceeds the threshold as by step Sc6. If the sound pressure SPis less than the threshold TH, the answer is given negative “NO”. Then,the data processing unit 704 changes the target frequency, and returnsto step Sc4. Thus, the data processing unit 356 changes the targetfrequency, and reiterates the loop consisting of steps Sc4 to Sc6 untilthe answer at step Sc6 is changed to affirmative.

When the data processing unit 704 find the peak P11(see FIG. 31), theanswer is changed to affirmative “YES”, and the data processing unit 704determines that the certain frequency is the standard pitch as by stepSc7. In the example shown in FIG. 31, the standard pitch is 443 Hz.

Subsequently, the data processing unit 704 acquires the discidentification code C-ID from the hard disc unit 414, and informs thefloppy disc player 606 of the standard pitch and disc identificationcode C-ID as by step Sc8. The floppy disc player 606 stores the eventcode representative of the standard pitch and disc identification codeC-ID in the floppy disc FD as shown in FIG. 32. In case where thestandard MIDI file SMF is to be created in the floppy disc FD, the eventcode representative of the standard pitch and disc identification codeC-ID are stored in the header chunk HT1.

The data processing unit 704 checks the manipulating panel 130 to seewhether or not the user has instructed the controller 702 to record theperformance on the keyboard 142 synchronously with the picture andfirst/second electronic tones as by step Sc9. If the answer at step Sc9is given negative “NO”, the data processing unit 704 returns to the mainroutine. On the other hand, when the answer is given affirmative “YES”,the data processing unit 704 informs the user that the multimediaplatform 700 gets ready for the synchronous recording, and instructs thefloppy disc recorder 414 to record the performance on the keyboard 142synchronously with the picture and first/second electronic tones as bystep Sc10. Upon completion of the performance, the user instructs thedata processing unit 704 to terminate the synchronous recording, and thedata processing unit 704 returns to the main routine.

The user is assumed to instruct the controller 704 for reproducing theperformance and second electronic tones synchronously with the pictureand first electronic tones. The data processing unit 704 enters asub-routine program shown in FIG. 33. First, the data processing unit704 requests the floppy disc player 606 to read out and transfer theevent code representative of the standard pitch from the floppy disc FDthereto as by step Sd1.

Subsequently, the data processing unit 704 instructs the manipulatingpanel 130 to produce a massage such as for example, “Please depress thewhite key A” on the display window, and waits for the user's response.When the user depresses the white key A, the hammer 146 strikes thestring 148, and the tone A is generated from the vibrating string 148. Amicrophone (not shown) picks up the tone A, and supplies the electricsignal to the data processing unit 704. The data processing unit 704analyzes the digital codes, which were converted from the electricsignal, and determines the standard pitch as by step Sd2. In thisinstance, the standard pitch for the piano tones is assumed to be 448Hz. The data processing unit 704 may measure the sound pressure level ina certain band through the fast Fourier transformation, and checks thesound pressure to see what frequency has the sound pressure level over athreshold. When the data processing unit 704 finds the sound pressurelevel at a certain frequency to exceed the threshold, the dataprocessing unit 704 determines that the certain frequency is thestandard pitch at the piano tone “A”.

Subsequently, the data processing unit 704 calculates the differencebetween the standard pitch of the electronic tone “A” and the standardpitch of the piano tone “A”, and determines a pitch difference as bystep Sd3. In this instance, the standard pitch of the piano tone “A” is448 Hz, and the standard pitch of the electronic tone “A” is 443 Hz sothat the pitch difference is 5 Hz. The electronic tones are to beincreased in pitch by 5 Hz. Then, the data processing unit 704determines a target speed for reading out the disc-stored audio datacodes from the hard disc unit 414 as by step Sd4. The data readout speeddeeply concerns the pitch of tones as follows.

FIG. 34A to 34C show the relation between the target speed for readingout the disc-stored audio data codes and the pitch of the electronictones. Even though the disc-stored audio data codes are not changed, thewaveform of the audio signal representative of the electronic tone isvaried depending upon the target speed for reading out the disc-storedaudio data codes from the hard disc unit 414. When the audio data codesare read out from the hard disc unit 414 at the standard read-out speedVb, the audio signal has a waveform form A′ shown in FIG. 34A. If thedata read-out is accelerated, i.e., Vf>Vb, the waveform A′ is shrunk,and the audio signal has the waveform B′ as shown in FIG. 34B.Accordingly, the tone is sharp pitched. The pitch is increased to 448Hz. On the other hand, in case where the read-out speed is lowered,i.e., Vs<Vb, the waveform A′ is expanded, and the audio signal has thewaveform C′ as shown in FIG. 34C. Accordingly, the pitch of the tone islowered to 440 Hz.

In this instance, the pitch of the second electronic tone is lower thanthe pitch of the acoustic piano tone by 5 Hz. The data processing unit704 instructs the hard disc unit 414 to increase the data read-outspeed. If, on the contrary, the pitch of the second electronic tones ishigher than the pitch of the acoustic piano tones, the data processingunit 704 instructs the hard disc unit 414 to decrease the data read-outspeed.

When the target speed is determined, the data processing unit 704instructs the hard disc unit 414, player 116 and floppy disc player 606to start the synchronous playback under the pitch control as by stepSd5. The player 116 reads out the video data codes, tape-stored audiodata codes and video time codes from the videotape cassette VT, thefloppy disc player 606 reads out the delta-time codes and event codesfrom the floppy disc FD, and the hard disc unit 414 reads out thedisc-stored audio data codes from the hard disc at the target speed asby step Sd6. The video data codes, video time codes and tape-storedaudio data codes are supplied to the monitor display 112, code converter126 and the data processing unit 704, respectively. The monitor display112 reproduces a picture on the screen. The video time codes areconverted to the MIDI time codes, and the data processing unit 704regulates the lapse of time with the MIDI time codes−The data processingunit 704 further transfers the MIDI time codes to the floppy disc player606 for regulating the lapse of time with the MIDI time codes. Thesefunctions are similar to those of the seventh embodiment, and no furtherdescription is incorporated hereinafter for avoiding repetition.

When the number N′ of tempo clocks CT catches up the target number M,the floppy disc player 606 supplies the event codes through the dataprocessing unit 704 to the MIDI controller 150, and the acoustic pianotones are generated in the acoustic piano 136. The data processing unit704 converts the disc-stored audio data codes to the analog audiosignal, and supplies the analog audio signal to the sound system 106 forproducing the second electronic tones. Since the disc-stored audio datacodes are read out at the target speed, the second electronic tones aresharp pitched, and are well harmonized with the acoustic piano tones.Thus, the picture, acoustic piano tones and electronic tones arereproduced as by step Sd7.

Subsequently, the data processing unit 704 checks the videotape cassetteVT, hard disc unit 414 and floppy disc FD to see whether or not all thedata codes have been already read out therefrom as by step Sd8. Whilethe answer at step Sd8 is given negative “NO”, the data processing unit704 returns to step Sd5, and reiterates the loop consisting of steps Sd5to Sd8 until the answer at step Sd8 is changed to affirmative “YES”.When the last video data/video time/tape-stored audio data/MIDI datacodes/disc-stored audio data codes are read out from the videotapecassette VT, hard disc 352 and floppy disc FD, the answer at step Sd8 ischanged to affirmative, and the data processing unit 704 returns to themain routine.

As will be understood from the foregoing description, the multimediaplatform 700 implementing the eighth embodiment achieves the pitchcontrol between the second electronic tones and the acoustic piano tonesas well as the synchronous playback. As a result, the electronic tonesare well harmonized with the acoustic piano tones.

Although particular embodiments of the present invention have been shownand described, it will be apparent to those skilled in the art thatvarious changes and modifications may be made without departing from thespirit and scope of the present invention.

In case where the multimedia platform 100 is expected to only record aperformance synchronously with a picture, the automatic player piano 138may be replaced with an electronic keyboard or a composite keyboardinstrument equipped with the key/pedal sensors 154/156. Neithersolenoid-operated key/pedal actuators 152/158 nor tone generators134/140 are required for the composite keyboard instrument. Moreover,the MIDI controller 150 is simplified. Thus, the multimedia platform forthe synchronous recording is much simpler than the multimedia platform100.

In the first embodiment, the performance on the keyboard 142 is recordedsynchronously with the playback of the picture. A first modification ofthe first embodiment records the performance on the keyboard 142synchronously with shooting the scene with the video camera 102. Theaccompaniment on another musical instrument such as, for example aviolin may be further recorded in the videotape VT. While theperformance is being taken with the video camera 102, the video recorder120 and sound recorder 122 writes the video data codes and audio datacodes in the videotape VT together with the video time codes, and thevideo time codes are supplied to the code converter 126. The codeconverter 126 converts the video time codes to the MIDI time codes, andthe data processing unit 128 supplies the MIDI time codes and MIDI eventcodes to the controller 172. The controller 172 behaves as similar tothe synchronous recording described in conjunction with the firstembodiment, and stores the MIDI event codes in the track chunk TTtogether with the delta-time codes. Thus, the first modification issuitable for a live concert and promotion disc.

The sound source 108 may have another sorts of musical instruments suchas, for example, electronic stringed instruments and electronic windowinstruments. The electronic stringed instruments are equipped withpickup units for converting the vibrations of strings to electricsignals, and music data codes are produced from the electric signals.The electronic wind instruments have key/piston monitors for convertingthe key actions/piston actions to electric signals, and music data codesare produced from the electric signals. The automatic player piano 136is replaceable with those electronic musical instruments. A personalcomputer, in which a suitable music composing program is installed, isalso available for the multimedia platform. The user produces MIDI eventcodes through a keyboard or mouth with the assistance of the software,and the personal computer supplies the MIDI event codes to thecontroller 110.

The floppy disc FD and videotape cassette VT do not set any limit on thepresent invention. Magneto optical discs, hard discs and memory sticksare available for the synchronous recording.

The controller 110, disc recorder/player 104, sound system 106 and tonegenerator for ensemble 134 may be built in the automatic player piano132, and the video camera 102 may be connected to the built-incontroller 110 through the cable. Thus, the multimedia platform 100according to the present invention is offered in the form of anautomatic player piano.

The disc player 308 may record the performance expressed by the MIDIdata codes in the floppy disc FD synchronously with the recording thevideo data codes, audio data codes and video time codes in the videotapecassette VT. While the image pickup device and microphone are supplyingthe video signal and audio signal to the video recorder 120 and soundrecorder 122, the time code generator 114 periodically supplies thevideo time codes to the videotape cassette VT and code converter 126.The code converter 126 converts the video time codes to the MIDI timecodes, and the data processing unit 310 transfers the MIDI time codes tothe disc player 308. The disc player 308 intermittently transfers theevent codes m[k] through the data processing unit 310 to the automaticplayer piano 132, and corrects the delta-time codes upon reception ofthe MIDI time codes as similar to the synchronous playback.

The multimedia platform 300 may be sold as an automatic player pianowith built-in video camera/controller/disc recorder/player 102/304/104.Of course, the video camera 102 is connected to the controller 304through a suitable cable so that the user arbitrarily directs the imagepickup device and microphone to an object.

In the fourth and eighth embodiments, the data processing units 356/704find the standard pitch for the electronic tones. However, the user mayhear the electronic tone for judging the standard pitch. In thisinstance, the user informs the data processing units 356/704 of thestandard pitch through the manipulating panel 130.

In the fourth and eighth embodiments, the data processing units 356/704make the manipulating panels 130 to transfer the message such as, forexample, “please depress the key A” to the user. In a modification ofthe fourth and eighth embodiments, the data processing units 356/704 maysupply a note-on event code for producing the tone “A” to the MIDIcontroller 150. The driving current is supplied form the driver circuit312 to the solenoid-operated key actuator associated with the white key“A” so that the microphone picks up the tone “A”.

The hard disc 352 may be replaced with another sort of memory such as asemiconductor random access memory. While the player 116 is reading outthe video data codes and video time codes from the videotape cassetteVT, the player 116 further reads out the audio data codes from thevideotape cassette VT, and transfers the audio data codes to thesemiconductor random access memory. The player 116 may intermittentlysupply the audio data codes to the semiconductor random access memorybefore the reproduction of the electronic tones. This results inreduction of memory capacity.

The controller 412 may check the music data code representative of thetitle of the music composition for the synchronous playback as well asthe disc identification code.

In the first, second and fifth embodiments, the automatic player piano132 may be replaced with a silent piano, i.e., a composite keyboardmusical instrument including an acoustic piano, key sensors, pedalsensors, a tone generator for piano tones 140 and a hammer stopper and aMIDI controller. Although the silent piano is not equipped with thesolenoid-operated key/pedal actuators, the MIDI controller analyzes thekey position signals and pedal position signals for producing the eventcodes. The event codes are supplied from the data processing unit to thedisc recorder so that the performance is recorded in a floppy discsynchronously with a picture/electronic tones or a picture/firstelectronic tones/second electronic tones.

In the fifth embodiment, the performance on the keyboard 142 is recordedand the second electronic tones are reproduced synchronously with theplay-back of the picture and the reproduction of first electronic tones.A modification of the fifth embodiment records the performance on thekeyboard 142 and reproduces the second electronic tones synchronouslywith shooting the scene with the video camera 102. While the imagepickup device is shooting the scene, the video recorder 120 and soundrecorder 122 stores the scenes and sound in the videotape cassette VT,and the time code generator 124 stores the video time codes in thevideotape cassette VT and transfers the video time codes to the player116. The video time codes are converted to the MIDI time codes, and thedata processing unit 424 supplies the MIDI time codes to the floppy discrecorder 404. The data transfer from the compact disc CD to the harddisc 414, data read-out from the hard disc 414 and recording into thefloppy disc FD are similar to those of the fifth embodiment so thatdetailed description is omitted. Thus, the modification of the fifthembodiment records the picture and performance in the videotape cassetteVT and floppy disc FD and reproduces the second electronic tonessynchronously with the shooting the scenes. The modification is suitablefor a live concert, promotion disc and video contents for shows.

The sound source 108 incorporated in the multimedia platform 400 mayhave another sorts of musical instruments such as, for example,electronic stringed instruments and electronic window instruments. Theelectronic stringed instruments are equipped with pickup units forconverting the vibrations of strings to electric signals, and music datacodes are produced from the electric signals. The electronic windinstruments have key/piston monitors for converting the keyactions/piston actions to electric signals, and music data codes areproduced from the electric signals. Acoustic musical instruments may beequipped with music code generating systems for producing music datacodes. The automatic player piano 132 is replaceable with thoseelectronic musical instruments. A personal computer, in which a suitablemusic composing program is installed, is also available for themultimedia platform. The user produces MIDI event codes through akeyboard or mouth with the assistance of the software, and the personalcomputer supplies the MIDI event codes to the controller 110. Thus, thesound source 408 stands for all the device, unit and system forproducing music data codes such as, for example, the MIDI data codes.

The floppy disc FD, compact disc CD, hard disc 414 and videotapecassette VT do not set any limit on the present invention. The hard disc414 is, by way of example, replaceable with magneto optical discs,semiconductor memory devices or memory sticks. In case where thesemiconductor random access memory is used for storing the disc-storedaudio data codes, compact disc controller/driver 420 may stepwisetransfer the disc-stored audio data codes to the data processing unit424, and the data processing unit 424 intermittently writes thedisc-stored audio data codes into and reads out them from thesemiconductor random access memory. For this reason, a large memorycapacity is not required for the semiconductor random access memory.

The controller 412, floppy disc recorder 404, sound system 406, compactdisc unit 410 and tone generator for ensemble 134 may be built in theautomatic player piano 132, and the video camera 102 may be connected tothe built-in controller 412 through a suitable cable. Thus, themultimedia platform 400 according to the present invention is offered inthe form of an automatic player piano.

A modification of the seventh embodiment may reproduce the acousticpiano tones and second electronic tones synchronously with recording thescenes in a videotape cassette VT. While the image pickup device isshooting the scenes, the video recorder 120 and sound recorder 122stores the visual images and sound in the videotape cassette VT, and thetime code generator 124 stores the video time codes in the videotapecassette VT and transfers the video time codes to the player 116. Thevideo time codes are converted to the MIDI time codes, and the dataprocessing unit 608 supplies the MIDI time codes to the floppy discplayer 606. The data transfer from the compact disc CD to the hard disc414, data read-out from the hard disc 414 and data read-out from thefloppy disc FD are similar to those of the seventh embodiment so thatdetailed description is omitted. Thus, the modification of the fifthembodiment records the picture and performance in the videotape cassetteVT and floppy disc FD and reproduces the second electronic tonessynchronously with the shooting the scenes. The modification is suitablefor a live concert, promotion disc and video contents for shows.

In the seventh embodiment, the disc-stored audio data codes have beenstored in the hard disc 414 before the synchronous playback. Anothermodification of the seventh embodiment may automatically store thedisc-stored audio data codes in the hard disc 414. In this instance,when a user instructs the video camera 402 to find the picture assignedthe video identification code V-ID, the data processing unit 608transfers the video identification code V-ID to the floppy disc player606, and instructs the floppy disc layer 606 to search the header chunkHHI whether or not a set of MIDI data codes assigned the videoidentification code V-ID has been already stored in the floppy disc FD.If the floppy disc player 606 finds the set of MIDI data codes assignedthe video identification code V-ID, the data processing unit 608instructs the floppy disc player 606 to read out and transfer theassociated disc identification code C-ID from the header chunk HTI. Thedata processing unit 608 transfers the disc identification code C-ID tothe compact disc unit 410, and instructs the compact disccontroller/driver 420 to read out and transfer the disc-stored audiodata codes labeled with the disc identification code C-ID. Thedisc-stored audio data codes are transferred to the data processing unit608, and the data processing unit 608 stores the disc-stored audio datacodes in the hard disc unit 414. The modification behaves as similar tothe seventh embodiment after the data write-in into the hard disc 414,and no further description is incorporated hereinafter.

The controller 604, floppy disc recorder/player 602, sound system 406,compact disc unit 410 and tone generator for ensemble 134 may be builtin the automatic player piano 132, and the video camera 102 may beconnected to the built-in controller 412 through a suitable cable. Thus,the multimedia platform 600 according to the present invention isoffered in the form of an automatic player piano.

A modification of the eighth embodiment may further control the pitch ofthe first electronic tones. In this instance, another hard disc isfurther connected to the data processing unit 704, the tape-stored audiodata codes are transferred to the other hard disc, and data processingunit 704 determines a target read-out speed for the tape-stored audiodata codes as similar to that for the disc-stored audio data codes.While the MIDI data codes are being read out from the floppy disc FD,the tape-stored audio data codes and disc-stored audio data codes areread out from the hard disc units at the individual read-out speeds.This results in harmonization among the acoustic piano tones, firstelectronic tones and second electronic tones.

In another modification of the eighth embodiment, the hard disc unit 414may be shared between the disc-stored audio data codes and the tapestored audio data codes as shown in FIG. 35. A memory area is assignedto the disc-stored audio data codes, and another memory area is assignedto the tapestored audio data codes. The target speed for the tape-storedaudio data codes is determined independently of the target speed for thedisc-stored audio data codes. While the player 116 is transferring thevideo data codes to the monitor display 112, the data processing unit704 reads out the disc-stored audio data codes at the target speed andthe tape-stored audio data codes at the other target speed, and thefloppy disc player 606 transfers the event codes through the dataprocessing unit 704 to the MIDI controller 150. Thus, the modificationreproduces the acoustic piano tones and second electronic tonessynchronously with the picture and first electronic tones, and makes thepiano tones, first electronic tones and second electronic tones wellharmonized with one another through the pitch control.

A multimedia platform according to the present invention may reproducethe second electronic tones through the pitch control synchronously withreproduction of a picture and first electronic tones and reproduction ofacoustic piano tones.

1. A multimedia platform for recording at least first music sounds in aninformation storage medium synchronously with a picture, comprising: afirst data source producing a first sort of data containing pieces offirst music data information representative of said first music sounds;a second data source producing a second sort of data containing piecesof video data information representative of visual images of saidpicture, pieces of first time data information representative of a firsttime defined from a first viewpoint, and a first time signalrepresentative of said pieces of first time data information; a thirddata source having an information processing capability and connected tosaid second data source, said third data source executing a computerprogram so as to realize a plurality of functions, said third datasource including a clock incrementing a second time defined from saidfirst viewpoint and represented by pieces of second time datainformation and generating a second time signal representative of saidpieces of second time data information, a comparator supplied with saidfirst time signal and said second time signal that compares said piecesof second time data information with said pieces of first time datainformation to determine whether said second time is consistent withsaid first time, a modifier modifying said pieces of second time datainformation with the negative answer so as to eliminate a timedifference between said first time and said second time, and a converterconverting said pieces of second time data information to pieces ofthird time data information representative of a third time defined froma second viewpoint different from said first viewpoint; a recorderconnected to said first data source and said third data source so as tostore said pieces of first music data information and said pieces ofthird time data information in said information storage medium; and animage generator connected to said second data source for producing saidvisual images, wherein said first data source includes a musicalinstrument on which said first music sounds are specified and a codegenerating system monitoring said musical instrument for generating saidpieces of first music data information specified through said musicalinstrument, wherein said second data source includes a video camera,wherein said musical instrument is an acoustic piano, and said pieces offirst music data information are produced through a fingering on akeyboard of said acoustic piano, and wherein said third data source,said recorder and said image generator form parts of an automatic playerpiano based on said acoustic piano, and said video camera is connectedto said third data source and said image generator through a cable. 2.The multimedia platform as set forth in claim 1, in which said pieces offirst time data information expresses a first lapse of time along whichsaid pieces of video data information are supplied to said imagegenerator, said pieces of second time data information expresses asecond lapse of time from an initiation of the recording, and saidpieces of third time data information expresses time intervals to beinserted among said pieces of first music data information.
 3. Themultimedia platform as set forth in claim 1, further comprising a soundgenerator connected to said second data source, in which said secondsort of data further contains pieces of second music data informationrepresentative of third music sounds and supplied to said soundgenerator concurrently with said pieces of video data information sothat said second music sounds are radiated from said sound generatorsynchronously with said picture.
 4. The multimedia platform as set forthin claim 3, in which said pieces of first time data informationexpresses a first lapse of time along which said pieces of video datainformation are supplied to said image generator, said pieces of secondtime data information expresses a second lapse of time from aninitiation of the recording, and said pieces of third time datainformation expresses time intervals to be inserted among said pieces offirst music data information.
 5. The multimedia platform as set forth inclaim 1, further comprising a sound generator, in which said second sortof data further contains pieces of second music data informationrepresentative of second music sounds and supplied to said soundgenerator for producing said second music sounds.
 6. The multimediaplatform as set forth in claim 5, further comprising a temporary datastorage connected to said first data source for storing said pieces ofsecond music data information and a read-out speed controller associatedwith said temporary data storage and reading out said pieces of secondmusic data information at a target speed for transferring said pieces ofsecond music data information to said sound generator, said target speedbeing determined in such a manner as to eliminate a pitch differencefrom between said first music sounds and the corresponding second musicsounds.